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ABSTRACT 

This  report  describes  the  development  and  psychometric  evaluation  of  the  Army  Learning 
Organisation  Questionnaire  (ALOQ).  Following  a  review  of  social  sciences  literature,  four 
questionnaires  used  to  measure  learning  organisation  characteristics  were  identified,  and  then 
evaluated  against  specific  validity  and  reliability  criteria.  Two  instruments  had  the  best 
psychometric  properties:  Marsick  and  Watkins'  (2003)  Dimensions  of  a  Learning  Organization 
(DLOQ)  and  Goh  and  Richard's  (1997)  Organisation  Learning  Survey  (OLS).  These  two 
questionnaires  were  adapted  for  use  within  the  Army  using  a  three  phase  pilot  process.  The 
resulting  questionnaire  was  named  the  ALOQ.  The  ALOQ  was  administered  to  the  target 
population  (390  Army  personnel)  and  its  psychometric  properties  were  evaluated.  The  ALOQ 
statistical  reliability  was  found  to  be  acceptable;  Cronbach  alphas  were  found  to  be  over  0.7 
for  dimensions.  ALOQ  validity  was  evaluated  using  both  theoretical  and  statistical  means.  An 
exploratory  factor  analysis  (N=3700  Army  personnel)  found  latent  factors  in  line  with  those 
found  in  DLOQ  and  OLS.  Hence  the  ALOQ  was  found  to  be  a  reliable  and  valid  measure  of 
learning  organisation  characteristics,  suitable  for  administration  within  the  military. 
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The  Army  Learning  Organisation  Questionnaire: 
Developing  valid  and  reliable  measure  of  Learning 
Organisation  characteristics 

Executive  Summary 

This  report  discusses  the  development  of  the  Army  Learning  Organisation  Questionnaire 
(ALOQ).  Learning  at  the  individual,  team,  and  organisational  levels  determines  an 
organisation's  capacity  to  generate,  share  and  exploit  useful  information.  Learning  within 
and  by  organisations  is  not  merely  a  'nice  to  have'  -  it  is  critical  to  improving 
organisational  performance.  Individual,  team  (or  collective)  and  organisational  level 
learning  have  all  been  linked  to  improved  organisational  performance. 

Yet  the  ability  (and  capacity)  to  learn  (at  individual,  team  and  organisation  levels)  is  not 
equally  visible  to  decision-makers.  Developing  and  sustaining  the  learning  capacity  across 
an  organisation  requires  decision-makers  to  be  informed  about  these  capabilities.  The 
Army  has  stated  its  aspiration  to  be  "a  Learning  Organisation";  consequently  the  ALOQ 
was  developed  in  order  to  provide  this  information  to  key  Army  decision-makers.  The 
ALOQ  aimed  to  be  a  valid  and  reliable  measure  providing  valuable  and  practical  data  on 
Army's  strengths  and  weaknesses  in  this  area.  Thus,  the  ALOQ  was  designed  to  provide 
critical  data  to  senior  Army  personnel  to  inform  their  decision-making. 

The  study  sought  to  answer  the  following  questions: 

•  What  is  the  reliability  and  validity  of  currently  existing  instruments  for  measuring 
Learning  Organisation  characteristics? 

•  Can  these  instruments  be  usefully  and  meaningfully  applied  to  the  Australian 
Army  context? 

Creating  quantitative  measures  of  social  phenomena  requires  developing  meaningful, 
concrete  measures  from  the  conceptual  and  theoretical  construct  (for  example,  a  Learning 
Organisation  would  include  information  sharing,  open  questioning  etc.).  These  concrete 
measures  should  be  specific  and  unambiguous,  with  logical  links  to  the  theoretical 
construct.  When  developing  psychometric  measurements,  the  measures  should  be 
evaluated  for  reliability  and  validity.  A  reliable  measure  is  consistent  over  time,  across 
populations  and  situations.  Validity  is  the  extent  to  which  the  measure  measures  the 
intended  phenomena,  for  example,  to  what  extent  can  inferences  can  be  drawn  from  the 
results  of  the  measure.  Reliability  is  a  necessary  but  not  sufficient  condition  to  ensure 
validity.  Reliability  and  validity  can  be  assessed  using  both  theoretical  and  statistical 
analysis. 
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A  systematic  literature  review  for  measures  of  Learning  Organisations  was  conducted, 
finding  four  instruments.  These  instruments  were  evaluated  against  well-established 
criteria  of  reliability  and  validity.  Of  the  four  instruments,  only  Marsick  and  Watkins' 
(2003)  Dimensions  of  Learning  Organisation  Questionnaire  (DLOQ)  and  Goh  and 
Richard's  Organisation  Learning  Survey  (OLS)  had  published  acceptable  levels  of 
reliability  and  validity1,  and  hence,  were  selected  to  be  the  basis  for  the  ALOQ.  The  other 
two  instruments  were  not  found  to  have  published  levels  of  reliability  and  validity,  or  did 
not  provide  permission  for  their  use  outside  the  author's  own  website. 

The  DLOQ  and  OLS  were  adapted  to  the  Army  in  order  to  develop  an  instrument,  which 
was  termed  the  ALOQ.  Development  included  an  iterative  pre-test  phase  with  Army 
personnel.  The  subsequent  statistical  analysis  (N=390  Army  personnel)  found  that  the 
ALOQ  had  acceptable  level  of  reliability  with  all  dimensions  being  above  the  acceptable 
level  of  Cronbach's  alpha  0.7.  An  exploratory  factor  analysis  (EFA)  was  conducted  to 
evaluate  the  statistical  validity  (N=3700  Army  personnel);  the  results  of  the  EFA  showed 
that  similar  factors  emerged  from  the  ALOQ  items  compared  to  with  the  DLOQ  and  OLS. 
There  was  one  small  difference  between  the  ALOQ  latent  factors  and  those  found  in  the 
DLOQ;  the  ALOQ  EFA  found  eight  factors,  compared  to  the  seven  found  by  Marsick  and 
Watkins'  DLOQ.  The  DLOQ  original  factor  continuous  learning  opportunities  was  found  to 
load  onto  two  separate  ALOQ  factors;  rewards  and  resources  and  learning  practices.  This 
means  that,  for  the  Army,  the  organisational-level  aspects  of  learning  (resourcing)  were 
not  directly  influencing  the  actions  and  practices  that  support  learning  within  a  workplace 
or  unit.  The  EFA  found  that  Goh  and  Richard's  (1997)  four  OLS  factors  were  replicated 
within  the  ALOQ  data.  This  means  that  there  was  a  similar  underlying  factor  influencing 
the  Army's  ALOQ  results  as  found  within  Goh  and  Richard's  (1997)  sample.  These  results 
support  the  theoretical  model  of  learning  organisations  developed  by  Marsick  and 
Watkins'  (2003)  and  Goh  and  Richard's  (1997). 

In  summary,  the  ALOQ  was  found  to  have  satisfactory  statistical  levels  of  reliability 
within  its  target  population.  The  EFA  of  the  ALOQ  show  alignment  with  the  original  work 
of  Marsick  and  Watkins'  (2003)  and  Goh  and  Richard's  (1997)  supporting  their  theoretical 
models  of  learning  organisations.  Consequently,  the  ALOQ  is  assessed  as  valid  and 
reliable  means  of  measuring  the  learning  organisational  characteristics  practiced  within 
the  Army. 


1  Acceptable  level  of  reliability  set  as  achieving  0.70  Cronbach  alpha  as  a  minimum  (Nunnally,  1978; 
Pallant,  2007;  Steiner  &  Norman,  2003).  Validity  is  assessed  using  nomological  network  and  its 
relationship  to  item  construction  using  theoretical  constructs  and  relationships  (Babbie  1995;  Clark 
and  Watson,  1995;  Streiner  and  Norman,  2003). 


UNCLASSIFIED 


UNCLASSIFIED 

DSTO-TN-1325 

Contents 

ACRONYMS 

1.  INTRODUCTION . 1 

2.  BACKGROUND . 1 

2.1  Psychometric  Measures . 1 

2.1.1  Key  Concepts  in  Psychometrics:  Reliability  and  validity . 3 

2.1.2  Reliability . 3 

2.1.3  Validity . 4 

2.1. 3.1  Content  Validity . 5 

2.1.4  Summary . 5 

2.2  The  Learning  Organisation . 6 

2.2.1  Measuring  the  Learning  Organisation . 7 

2.2.2  Research  Questions . 8 

3.  METHOD . 8 

3.1  Evaluation  of  Learning  Organisation  Measures . 9 

3.2  Validity  of  Learning  Organisation  Measures . 9 

3.2.1  Conceptualisation . 9 

3.2.2  Nominal  Definition . 9 

3.2.3  Operational  Definition . 10 

3.3  Reliability  of  Learning  Organisation  Measures . 12 

4.  RESULTS:  DEVELOPING  THE  ARMY  LEARNING  ORGANISATION 

QUESTIONNAIRE . 14 

4.1  ALOQ  Pre-test  and  Testing  Process . 15 

4.1.1  First  Pretest:  In-house  DSTO  Army  Personnel . 15 

4.1.2  Second  Pre-test:  Land  Warfare  Development  Centre  Workshop  - 

Army  Personnel . 15 

4.1.3  Third  Pre-test:  Soldier  Interviews . 16 

4.2  ALOQ  Reliability  and  Validity . 16 

4.2.1  ALOQ  Reliability . 16 

4.2.1. 1  ALOQ  Internal  Consistency . 16 

4.2.2  ALOQ  Validity . 17 

4.2.2.1  ALOQ  Content  and  Face  Validity . 17 

4.3  ALOQ  Statistical  Analysis . 18 

4.3.1  Participants . 18 

4.3.2  Treatment  of  missing  data . 18 

4.3.3  Exploratory  factor  analysis  (EFA)  of  ALOQ . 18 

4.3. 3.1  Checking  correlations  and  commonalities . 19 

4.3. 3.2  ALOQ  Factor  extraction.  Eigen  Values,  Scree  Plots  and  the  Factor 

Matrix . 20 

4.4  Summary  of  ALOQ  reliability  and  validity . 23 


UNCLASSIFIED 


UNCLASSIFIED 

DSTO-TN-1325 

5.  CONCLUSIONS . 25 

6.  REFERENCES . 26 

APPENDIX  A:  ALOQ  EFA  RESULTS . 31 

A.l.  ALOQ  items . 31 

A.2.  ALOQ  communalities . 32 

A.3.  EFA  results . 33 


UNCLASSIFIED 


UNCLASSIFIED 


DSTO-TN-1325 


Acronyms 


ALE 

Army  Learning  Environment 

ALO 

Army  Learning  Organisation 

ALOQ 

Army  Learning  Organisation  Questionnaire 

CPL 

Corporal 

DF 

Degrees  of  Freedom 

DLOQ 

Dimensions  of  a  Learning  Organization  Questionnaire 

EFA 

Exploratory  Factor  Analysis 

FIML 

Full  Information  Maximum  Likelihood 

KMO 

Kaiser-Meyer-Oklin 

LEQ 

Learning  Environment  Questionnaire 

LO 

Learning  Organisation 

LOS 

Learning  Organization  Survey 

LWDC 

Land  Warfare  Development  Centre 

LTCOL 

Lieutenant  Colonel 

ML 

Maximum  Likelihood  Extraction  Method 

NCO 

Non-commissioned  Officers 

OLS 

Organisational  Learning  Survey 

OR 

Other  Ranks 

PC 

Principle  Component  Extraction  Method 

SPSS 

Statistical  Package  for  the  Social  Sciences 

WOl 

Warrant  Officer  1 

W02 

Warrant  Officer  2 

UNCLASSIFIED 


DSTO-TN-1325 


UNCLASSIFIED 


Tltis  page  is  intentionally  blank 


UNCLASSIFIED 


UNCLASSIFIED 


DSTO-TN-1325 


1.  Introduction 


This  report  describes  the  development  and  psychometric  evaluation  of  the  Army  Learning 
Organisation  Questionnaire  (ALOQ).  The  ALOQ  has  been  designed  to  measure 
organisational  characteristics  that  are  known  to  support  learning  within,  and  by, 
organisations.  Learning  at  the  individual,  team,  and  organisational  levels  all  contribute  to 
an  organisation's  capacity  to  generate,  share  and  exploit  information.  Yet  the  capacity  to 
learn  is  not  necessarily  readily  (or  equally)  visible.  The  Army  has  stated  its  aspiration  to  be 
"a  Learning  Organisation"  (Australian  Army,  2007).  Developing  a  valid  and  reliable 
instrument  to  measure  the  Army's  learning  capabilities  allows  us  to  capture  valuable  and 
practical  data  that  can  then  be  used  by  senior  Army  decision-maker  and  stakeholders  to 
inform  their  decision-making.  Understanding  Army's  current  capability  to  learn  is  a  vital 
step;  using  this  knowledge,  decision-makers  can  then  decide  how  to  best  apply  and  exploit 
their  limited  resources  (including  effort,  and  attention)  to  maximise  learning.  Developing  a 
measure  of  the  Army's  learning  capability  presents  critical  information  for  improving 
organisational  performance  and  evaluating  the  impact  of  proposed  organisational 
initiatives. 

Initially,  this  report  describes  the  application  of  social  sciences  principles.  The  report  then 
briefly  reviews  the  literature  on  organisational  learning.  The  available  organisational 
learning  survey  instruments  were  critiqued  against  the  two  most  important  psychometric 
criteria;  validity  and  reliability.  The  best  of  the  available  instruments  were  adapted  to 
apply  to  the  Army,  producing  the  ALOQ;  the  adaptations  and  modifications  of  these 
questionnaires  are  described.  The  final  part  of  the  report  evaluates  the  ALOQ  using 
psychometric  criteria  of  validity  and  reliability  and  it  also  discusses  the  utility  of  the 
ALOQ  to  the  Army. 


2.  Background 


2.1  Psychometric  Measures 

Babbie  contends  that  the  social  sciences  measure  "the  stuff  of  life;  love,  hate,  prejudice, 
radicalism,  [and]  alienation"  (1995,  p.  110).  Social  science  offers  a  way  of  measuring  "the 
stuff  of  life"  by  observing  and  quantifying  hypothesised  concepts  by  applying  the 
following  process  (see  Figure  1). 
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Conceptualisation: 

Description  of  the  phenomena 

J 

Nominal  definition: 

Assigned  and  specified  definition 

J 

- 

Operational  definition: 

Methodology  and  measurements 

L 

L 

Measurement 

Figure  1:  From  Conceptualisation  to  Measurement:  Quantifying  theoretical  constructs  (Babbie, 
1995). 


The  aim  of  this  process  is  to  make  a  workable  definition  of  the  phenomena  of  interest,  so 
that  it  is  specific  and  unambiguous;  whilst  somebody  may  disagree  with  the  definition, 
they  can  trace  how  the  measures  are  achieved  (Babbie,  1995).  There  is  frequently  a  tension 
between  the  precision  necessary  for  scientific  inquiry  and  the  contextual  information  that 
is  embedded  within  social  phenomena.  In  this  respect,  often  the  specification  needed  for 
operational  definitions  seem  to  rob  concepts  of  their  richness  and  meaning.  If  there  is  no 
single  clear  way  to  measure  a  concept,  the  use  of  multiple  methods  (including 
triangulation)  can  be  employed  to  build  up  a  richer  picture  of  the  concept.  Using  multiple 
methods  also  addresses  the  issues  associated  with  measurement  errors  (Podsakoff, 
MacKenzie,  Lee,  &  Podsakoff,  2003). 

Stevens'  definition  (1946:  p.677)  of  measurement  as  "the  assignment  of  numerals  to  objects 
or  events  according  to  some  rule"  is  the  accepted  definition  of  measurement  in  social 
sciences.  Measurement  can  be  classed  into  four  different  types:  "nominal"  scale  (or 
categorical),  "ordinal"  scale,  "interval"  scale  and  "ratio"  measurement.  The  type  of 
measurements  or  data  collected  determines  the  levels  of  analysis  that  can  be  conducted. 
For  example,  gender  is  measured  as  a  nominal  scale  and  hence,  can  only  be  analysed  using 
non-parametric  methods  of  analysis  or  used  as  a  means  of  grouping  the  sample  (e.g. 
comparing  males  to  females). 

The  use  of  the  word  "assignment"  within  the  concept  indicates  that  measurement  or 
quantification  is  arbitrary  or  subjective.  Developing  scales  to  measure  subjective  attributes 


2 


UNCLASSIFIED 


UNCLASSIFIED 

DSTO-TN-1325 

is  neither  easy  nor  simple  and  requires  "considerable  investment  of  both  mental  and  fiscal 
resources"  (Streiner  &  Norman  2008;  p5).  Developing  and  using  a  response  scale  which 
assigns  both  a  useful  and  meaningful  value  requires  a  systematic  approach.  Psychological 
and  educational  researchers  have  developed  a  comprehensive  set  of  standards  used  to 
assess  such  measures  (Streiner  &  Norman,  2008).  The  standards  were  set  out  in  a  manual 
(Standards  for  Educational  and  Psychological  Testing )  and  published  jointly  by  the  American 
Educational  Research  Association,  the  American  Psychological  Association  and  the 
National  Council  on  Measurement  in  Education  (AERA/  APA/NCME,  1999). 

2.1.1  Key  Concepts  in  Psychometrics:  Reliability  and  validity 

Psychometrics'  key  concepts  include  reliability  and  validity  (Campbell,  1960;  Carmines  & 
Zeller,  1979;  Streiner  &  Norman,  2003;  2008).  A  reliable  measure  is  one  that  measures  the 
consistency  of  a  construct  (over  time,  across  individuals  and  situations).  Validation  is  the 
extent  that  you  can  draw  inferences  from  the  test  (Streiner  &  Norman,  2003;  2008),  or  the 
degree  to  which  the  measurement  process  measures  the  construct  it  claims  to  measure 
(Gravetter  &Forzano,  2009).  Reliability  is  a  necessary  but  not  sufficient  condition  to  ensure 
validity.  Both  reliability  and  validity  can,  and  are,  assessed  statistically  (Streiner  &Norman 
2003,  2008;  Holmes-Smith,  2011). 

2.1.2  Reliability 

Reliability  is  "a  fundamental  way  to  reflect  the  amount  of  error,  random  and  systematic, 
inherent  in  any  measurement"  (Streiner  &  Norman,  2003;  p  126).  Streiner  and  Norman 
(2003)  describe  how  reliability  has  been  formally  based  on  classical  test  theory  and  simply 
state  that  any  observation  is  made  up  of  two  components:  a  true  score,  and  an  error 
associated  with  the  measurement.  The  reliability  formula,  called  intraclass  correlation,  was 
first  described  by  Fischer  (1925)  to  examine  the  extent  of  inter-rater  agreement.  A  number 
of  formulations  of  inter-rater  reliability  have  been  developed  including  test-retest 
correlations  (Pearson  correlation  coefficient)  to  compare  rater's  results  over  time  and 
across  populations  or  situations  (Streiner  &  Norman,  2003,  2008). 

A  self-completed  measure  has  a  different  source  of  error  when  compared  to  rater- 
completed  measures;  self -completed  measures  have  internal  consistency  of  a  measure  as  a 
source  of  error.  Internal  consistency  (the  homogeneity  of  a  single  test)  may  be  assessed 
using  several  difference  approaches;  item-total  correlation,  split-half  reliability  and 
Cronbach's  alpha.  The  item-total  correlation  is  "the  correlation  of  the  individual  item  with 
the  scale  total  omitting  the  item”  (Streiner  &  Norman,  2003,  p  70  italics  as  original).  The 
guide  for  establishing  a  useful  correlation  between  the  item  and  the  total  score  is  above 
0.20;  items  with  lower  correlations  should  be  discarded  (Kline,  1986).  Streiner  and  Norman 
(2003)  state  that  the  most  common  correlation  coefficient  is  the  Pearson  correlation 
coefficient,  as  it  is  sufficiently  robust  to  withstand  a  non-normal  distribution.  The  split-half 
reliability  is  calculated  by  randomly  dividing  into  two  sub-scales,  which  are  then 
correlated  with  each  other.  If  the  scale  is  consistent,  the  two  subscales  should  be  highly 
correlated. 
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The  most  commonly  used  index  of  reliability  is  Cronbach's  alpha  (Cronbach,  1951;  De 
Vellis,  2003;  Nunnally,  1978;  Pallant,  2007;  Streiner  &  Norman,  2003,  2008).  Cronbach's 
alpha  builds  on  the  utility  of  the  split-half  reliability,  and  also  allows  for  an  examination  of 
each  item's  contribution  to  the  overall  reliability  of  the  scale.  Cronbach's  alpha  give  the 
average  of  all  the  possible  split-halves  reliabilities  of  the  scale.  The  scale's  alpha  is 
calculated,  omitting  each  item  in  turn.  If  the  alpha  increases  when  a  specific  item  is 
removed,  this  indicates  that  the  homogeneity  or  internal  consistency  of  the  scale  is 
improved.  Streiner  and  Norman  (2003)  point  out  that  alpha  is  dependent  upon  both  the 
number  of  items  within  the  scale  and  the  magnitude  of  the  correlations  between  the  items. 
A  very  high  alpha  may  in  fact  be  generated  from  item  redundancy,  hence,  the  alpha 
should  be  above  0.70  but  normally  not  higher  that  0.90  (Streiner,  2003a). 

Reliability  is  not  merely  concerned  with  'agreement'  in  a  scale;  the  reliability  of  a  scale  is 
"intimately  linked  to  the  population  to  which  one  wants  to  apply  the  measure"  (Streiner 
and  Normal,  2003;  p  130).  Reliability  coefficients  only  have  meaning  when  applied  to  a 
specific  population  (Streiner,  2003a);  so  reliability  is  relative.  The  homogeneity  of  a 
population  will  determine  the  internal  consistency  of  the  measure,  for  example,  Streiner 
and  Norman  (2003)  give  an  example  of  how  the  variation  within  a  quality  of  life  scale 
would  be  very  different  if  given  to  a  homogenous  group  of  rheumatoid  patients  compared 
to  a  group  consisting  of  both  disease-free  or  ankylosing  spondylitis  patients.  The  reliability 
of  the  scale  will  be  greater  in  the  homogenous  group  than  the  heterogeneous  groups 
(Streiner,  2003a).  Therefore,  reliability  coefficients  need  to  be  calculated  anew  each  time 
the  measure  is  used  with  a  different  population. 

2.1.3  Validity 

Validity  is  defined  as  the  extent  to  which  evidence  and  theory  support  the  interpretations 
of  the  test  score  (AERA/APA/NCME,  1999);  that  is,  the  degree  of  confidence  placed  on 
the  measure  and  therefore,  the  higher  the  validity  of  a  measure,  the  greater  the  inferences 
that  can  be  drawn  under  a  variety  of  conditions  (Streiner  &  Norman,  2003,  2008).  Streiner 
and  Norman  (2003)  define  validity  as  "a  process  of  hypothesis  testing'  (p  174,  italics  as 
original)  such  that  validation  allows  inferences  on  the  results  to  be  drawn  from  the 
measures  of  a  particular  population  or  circumstance. 

Validity  can  be  assessed  in  a  variety  of  ways  (Weiner  &  Braun,  1988;  Cronbach  &  Meehl, 
1955;  Babbie,  1995;  Cook  &Beckman,  2006;  Streiner  &  Norman,  2003)  and  historically,  four 
types  have  been  defined: 

•  Construct  validity:  the  extent  to  which  the  practical  test  actually  measures  the 
phenomena,  for  example,  to  what  extent  do  IQ  questionnaires  measure  real  world 
intelligence? 

•  Content  validity:  the  extent  to  which  the  measure  covers  the  phenomena 
investigated,  for  example,  does  an  IQ  questionnaire  measure  all  aspects  of 
intelligence  discussed  within  the  scientific  literature? 
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•  Criterion  validity:  the  degree  of  correlation  between  the  measure  and  an 
established  measure  or  test  (the  criteria)  already  shown  to  be  valid,  for  example,  IQ 
tests  are  often  correlated  with  measures  of  academic  performance  (the  criterion). 

•  Predictive  validity:  the  extent  the  measure  correctly  forecasts  other  results  of  the 
same  construct  in  the  future.  For  example,  does  an  IQ  test  correctly  predict  the 
person's  future  academic  performance? 

•  Face  validity:  to  what  extent  does  a  test  appear  to  measure  the  behaviour  of 
interest.  For  sensitive  issues,  face  validity  may  generate  biases,  for  example,  faking 
good  or  faking  bad  to  make  the  respondents  appear  in  a  more  socially  desirable 
light. 

These  different  aspects  of  validity  are,  in  fact,  nested.  Content  validity  encompasses  all  of 
the  above  dimensions  of  validity  (Streiner  &  Norman,  2003,  2006).  The  extent  to  which  a 
measures  predicts  outcome  (predictive  validity)  or  is  correlated  to  a  related  outcome 
(criterion  validity)  rest  upon  the  inclusion  of  all  aspects  of  the  phenomena  (construct 
validity),  which  determines  the  extent  to  which  the  measure  can  be  used  to  infer 
consequences  (content  validity).  Current  thinking  in  the  area  focuses  on  all  aspects  of 
content  validity  as  a  holistic  approach  (Streiner  &  Norman,  2003;  2006). 

2.1. 3.1  Content  Validity 

Content  validity  is  the  degree  to  which  a  measure  encompasses  the  relevant  or  important 
aspects,  given  the  theoretical  grounding  of  the  construct  of  interest  (Babbie,  1995;  Streiner 
&  Norman,  2003,  2008).  Once  the  items  have  been  developed  by  applying  Babbie's  process 
from  conceptual  definition  to  an  operational  definition  and  then  concrete  measures, 
content  validity  refers  to  how  much  of  the  concrete  measures  cover  the  desired  aspects  of 
the  original  conceptual  definition.  Assessing  content  validity  can  be  done  by  assessing  the 
extent  to  which  the  operationally  defined  measures  match  all  aspects  of  the  conceptual 
definition.  Are  there  any  pertinent  factors  that  have  not  been  covered  within  the 
operational  definition,  and  are  there  any  irrelevant  aspects  included  within  the  operational 
definition  that  do  not  contribute  to  the  conceptual  definition?  This  comparison  needs  to  be 
done,  initially,  at  a  theoretical  level  as  it  requires  a  thorough  understanding  of  the 
conceptual  and  theoretical  basis  of  the  phenomenon  of  interest. 

Validity  can  be  evaluated  statistically,  as  well  as  theoretically.  Applying  an  exploratory 
factor  analysis  (EFA)  to  a  data  set  about  the  phenomena  of  interest  is  a  means  of  analysing 
the  underlying  structure  of  the  data;  EFA  is  a  means  of  establishing  if  the  empirical 
evidence  aligns  with  the  theorised  models.  If  the  theory  is  correct,  then  the  factors 
emerging  from  the  EFA  should  align  with  the  theorised  model. 

2.1.4  Summary 

In  order  to  assess  the  utility  of  any  questionnaire  (or  instrument,  measure,  scale)  the  key 
criteria  are  that  it  be  reliable  and  valid.  Reliability  means  that  the  instrument  is  measuring 
something  in  a  reproducible  fashion;  it  is  commonly  defined  as  the  ratio  of  the  variability 
between  the  individuals  to  the  total  variability  in  the  scores.  In  self-report  measures, 
internal  consistency  and  stability  are  key  aspects  of  reliability.  These  can  be  assessed 
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empirically  using  various  statistical  analyses  (Cronbach's  alpha  and  correlation, 
respectively).  Validity  is  the  extent  to  which  a  test  measures  the  real-world  phenomena 
and  needs  to  be  assessed  both  theoretically  and  empirically.  Although  there  are  many 
approaches,  they  can  be  grouped  into  either: 

•  Comparisons  of  the  measure  to  other  similar  scales /  measures  (and  establishing  the 
level  of  agreement  between  the  newly  developed  and  the  established).  The  level  of 
agreement  would  be  statistically  calculated  most  commonly  using  correlations, 
analysis  of  variance. 

•  Assessment  of  the  extent  to  which  the  newly  developed  scale  supports 
theoretically-expected  relationships  between  known  groups  or  differences. 

Validity  and  reliability  are  key  criteria  in  evaluating  psychometrically  measures;  a  valid 
and  reliable  measure  provides  useful  and  robust  information  that  would  otherwise  be 
unavailable. 

2.2  The  Learning  Organisation 

To  be  successful  in  a  competitive  environment,  organisations  need  to  foster  a  culture  for 
creating,  acquiring,  and  transferring  knowledge,  and  more  significantly,  modifying 
behaviour  (at  all  levels)  to  reflect  new  knowledge  and  insights  (Goh,  Elliot  &  Quon,  2012; 
Edmondson,  2004;  Edmondson,  Bohmar  &  Pisano,  2001;  Yang,  Marsick  &  Watkins,  2004). 
Learning  is  not  merely  a  'nice  to  have';  successful  learning  is  linked  to  improved 
organisational  performance  (summarised  in  Talbot,  Stothard,  Drobnjak  and  McDowall,  in 
preparation).  The  ability  to  generate,  capture,  share  and  act  on  new  knowledge  and 
insights  are  hallmarks  of  a  learning  organisation  (Garvin,  2008).  Indeed,  the  term  'learning 
organisation'  has  become  a  common  phrase  for  describing  a  host  of  approaches  to 
organisational  development  and  a  plethora  of  definitions  of  a  learning  organisation  exist 
(Talbot,  Drobnjak,  Stothard  and  McDowall,  in  prep;  Senge,  1990).  A  variety  of  mechanisms 
or  'building  blocks'  are  prescribed  for  ensuring  the  generation  of  cultural  conditions  that 
facilitate  learning  within  an  organisational  context.  Some  of  these  building  blocks  include 
providing  "strategic  leadership  for  learning",  "creating  continuous  learning 
opportunities",  "promoting  dialogue  and  inquiry",  and  "generating  a  shared  vision" 
(Marsick  &  Watkins,  2003;  Yang,  Watkins  &Marsick,  2004;  Goh  &  Richards,  1997). 

While  the  learning  organisation  construct  has  been  used  within  a  variety  of  organisations, 
very  little  attention  has  been  given  to  the  application  of  learning  organisation  practices 
within  military  institutions.  The  Australian  Army  faces  a  variety  of  unique  organisational 
learning  challenges  (Talbot,  Stothard,  Drobnjak  &McDowall,  in  prep;  Talbot,  Fidock, 
Drobnjak  &Stothard,  2007;  Talbot,  2006;  Talbot  &0'Toole,  2008;  Stothard  &Drobnjak, 
2009a;  2009b).  Newton  (2010)  found  that  Australian  Army  personnel's  experiences  of 
learning  organisation  characteristics  were  mainly  determined  by  the  organisational 
context.  Participants  reported  greater  levels  of  micro-management,  risk  averseness  and 
had  little  ability  to  engage  with  responsive  or  creative  work  when  in  barracks. 

A  summary  of  the  defining  features  of  learning  organisations  is  presented  in  Talbot, 
Stothard,  Drobnjak,  and  McDowall  (in  prep).  There  are  often  multiple  learning  theories 
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employed  within  each  definition  (i.e.  whether  the  definition  draws  on  behavioural, 
cognitive,  cultural,  structural  or  systemic  perspectives  etc.),  and  whether  the  definition 
refers  to  the  Learning  Organisation  as  an  aspirational  quality  or  type  of  organisation. 

2.2.1  Measuring  the  Learning  Organisation 

Despite  the  large  number  of  papers  describing  or  discussing  the  desirable  characteristics  of 
a  Learning  Organisation,  relatively  few  attempts  have  been  made  at  measuring  these 
characteristics.  Yet  an  understanding  of  the  current  situation,  that  is,  a  baseline  of  an 
organisation's  current  learning  capabilities,  is  an  essential  step  in  developing 
organisational  change  in  order  to  support  and  sustain  learning.  Consequently,  the  existing 
literature  on  measures  of  Learning  Organisations  was  reviewed  and  four  published 
instruments  were  found  (Table  1). 


Table  1:  Currently  available  Measures  of  Learning  Organisation  capabilities 


Instrument 

Authors/Source 

Availability/ 

Administration 

Number  of 
dimensions, 
response  scale 

Dimensions 

Dimensions  of  Learning 
Organization  Questionnaire 
(DLOQ) 

Marsick  &  Watkins 
(2003) 

Publically 

available 

7  dimensions, 

6  point  scale 

•  Continuous  learning 
practices 

•  Dialogue  and  inquiry 

•  Teamwork  and 
collaboration 

•  System  -  capture  and 
share  learning 

•  Collective  vision 

•  Connect  org.  to 
environment 

•  Strategic  leadership  for 
learning 

Organizational  Learning 
Survey  (OLS) 

Goh  &  Richards 
(1997) 

Publically 

available 

5  dimensions, 

7  point  scale. 

•  Clarity  of  mission, 

•  Teamwork 

•  Experimentation, 

•  Leadership 

•  Transfer  of  knowledge 

Learning  Environment 
Questionnaire  (LEQ) 

Armstrong  &  Foley 
(2003) 

Publically 

available 

2  factors  in  4  issues 

•  Mission  linked  learning, 

•  Facilitative  learning 
environment, 

•  Mission  support 

•  Learning  identification 
satisfaction 

•  Organisation  support. 

Low  personal  impact, 
mentoring  and  coaching. 

Learning  Organization 
Survey  (LOS) 

Garvin,  Edmondson 
&  Gino  (2008) 

Online  only 
through 

los.hbs.edu  -  not 

publically 

available 

3  Building  Blocks 
(dimensions)  each 
made  up  of  between 

1  to  5  scales  with  5 
items  each.  Scoring 
is  on  7  point  scale 

•  Supportive  learning 
environment 

•  Concrete  learning 
processes  and  practice, 

•  Leadership  that  reinforces 
learning. 
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2.2.2  Research  Questions 

The  four  questionnaires  are  critiqued  and  reviewed  in  order  to  provide  the  basis  for 
developing  a  measure  of  the  Army's  organisational  learning  capabilities.  The  development 
of  a  valid  and  reliable  instrument  to  measure  the  Army's  learning  capabilities  supports  the 
generation  of  useful  information  that  can  then  be  used  to  inform  Army  decision  makers. 
To  achieve  this,  the  following  questions  were  investigated: 

•  What  is  the  reliability  and  validity  of  existing  instruments  that  measure  Learning 
Organisation  characteristics? 

•  Can  these  instruments  be  applied  to  the  Army? 


3.  Method 


Developing  measures  of  subjective  experiences,  especially  at  the  organisational  level  (e.g.  a 
learning  culture)  requires  considerable  investment.  Streiner  &  Norman  (2003,  2008) 
extensively  reviewed  the  methods  for  developing  measures  of  subjective  attributes  or 
experiences.  They  describe  the  ideal  development  process  of  scale  development.  This 
process  has  been  applied  to  the  development  of  the  ALOQ;  the  theoretical  validity  of  the 
ALOQ  was  examined  using  the  nomological  process  described  by  Babbie  (1995)  and 
Streiner  and  Norman  (2003;  2008). Streiner  and  Norman  (2003)  describe  the  following 
process  as: 

1.  Find  any  existing  scales  or  measures  within  the  literature 

2.  Critically  review  measures 

a.  Reliability 

b.  Validity 

3.  Pre-test  measures  on  a  target  population  to  ensure  that  items  or  questions  are 

a.  Comprehensible 

b.  Unambiguous 

c.  Only  ask  a  single  question 

4.  Eliminate  or  rewrite  items  in  response  to  the  pre-test,  and  repeat  until  the  items 
fulfil  the  criteria  of  being  understandable,  clear  and  only  ask  a  single  question. 

5.  Check  for  internal  consistency  of  the  developed  measures  for  the  target  population: 

a.  Reliability  criteria 

b.  Validity  criteria. 

6.  Conduct  statistical  analysis  using  target  population  in  order  to  evaluate  the 
measures'  empirical  validity  using  Cronbach  alpha  and  exploratory  factor  analysis 
(EFA). 


8 


UNCLASSIFIED 


UNCLASSIFIED 


DSTO-TN-1325 


The  statistical  reliability  of  the  DLOQ  and  OLS  dimensions  were  analysed,  using  the 
software  package  IBM  SPSS  (Statistics  Package  for  the  Social  Sciences). 


3.1  Evaluation  of  Learning  Organisation  Measures 

A  review  of  the  open  literature  yielded  four  instruments  (Table  1).  Despite  the  large 
number  of  papers  describing  Learning  Organisations  (LO)  only  a  few  attempts  have  been 
made  at  measuring  their  characteristics.  All  four  papers  of  the  instrument  development 
shared  the  main  themes  found  within  the  LO  literature  (such  as  the  importance  of 
communication,  leadership,  teamwork,  information  processes  and  infrastructure,  and 
shared  engagement  in  generating  learning  capacity  within  organisations).  The  next  step 
was  to  examine  the  reliability  and  validity  of  these  instruments. 


3.2  Validity  of  Learning  Organisation  Measures 

Given  that  Streiner  and  Norman  (2003,  2008)  describe  how  validity  should  be  first  assessed 
on  a  theoretical  basis,  the  instruments  were  reviewed  using  Babbie's  (1995)  approach;  a 
process  of  explicitly  moving  from  the  conceptual  definition  towards  the  operational 
definition  as  a  framework.  Babbie  (1995)  posits  that  the  foundation  of  valid  measures  is 
based  in  the  theoretical  model  within  which  it  is  framed.  Thus,  all  measures  of  a  subjective 
experience  need  to  explicitly  link  the  hypothesised  underpinning  constructs. 

3.2.1  Conceptualisation 

There  are  a  number  of  formal  definitions  within  the  published  literature  on  what 
constitutes  a  learning  organisation  (see  Talbot  et  al  in  prep).  The  definitions  draw  on 
behavioural,  cognitive,  cultural,  structural  and  systemic  perceptions.  All  of  the 
instruments  reviewed  were  underpinned  by  specific  (but  differing)  definitions  of  a 
learning  organisation.  For  example  Marsick  and  Watkins  (2003;  p.  8)  emphasised  the 
behavioural  and  adaptive  aspects  of  their  definition:  a  learning  organisation  "learns 
continuously  and  transforms  itself".  In  a  similar  fashion,  Garvin  (1993;  p.  80)  focuses  on 
the  cognitive,  behavioural  and  adaptive  aspects  when  defining  a  learning  organisation. 
Goh  and  Richards  (1997;  p.  577)  use  a  normative  perspective,  describing  learning  as  a 
collective  activity  that  takes  place  under  particular  organisational  conditions.  Interestingly, 
Goh  and  Richards  adopt  Garvin's  (1993)  definition  of  a  learning  organisation;  "A  learning 
organisation  is  an  organisation  skilled  at  creating,  acquiring  and  transferring  knowledge 
and  at  modifying  its  behaviour  to  reflect  new  knowledge"  when  they  developed  the 
Organizational  Learning  Survey  (OLS). 

3.2.2  Nominal  Definition 

The  authors  of  all  four  instruments  developed  a  nominal  or  working  definition  for  a 
learning  organisation,  which  in  turn,  underpinned  the  operational  definition  (Marsick  & 
Watkins,  2003;  Garvin  &  Edmondson,  2008;  Goh  &  Richards,  1997,  Armstrong  &Foley, 
2003).  All  authors  defined  learning  organisations  as  a  complex,  multilayered  construct. 
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and  therefore,  have  used  multiple  dimensions  in  their  operational  definitions  as  Babbie 
(1995)  recommended.  The  nominal  definition  essentially  described  learning  organisations 
to  be  made  up  of  different  dimensions  (all  generated  from  a  theoretical  and  empirical 
basis),  and  the  common  aspects  include  (Talbot,  Drobnjak,  Stothard  &McDowall,  in 
preparation): 

•  learning  occurring  within  organisational  levels  or  subsystems 

•  organisational  culture  e.g.  norms,  attitudes,  accepted  practices 

•  systems  thinking 

•  teamwork 

•  communication 

•  leadership 

•  information  infrastructure 

•  engagement  with  shared  understanding  or  vision. 

Thus,  the  nominal  definition  of  a  learning  organisation  is  one  which  exhibits  specific 
aspects  or  characteristics,  for  example,  personnel  can  have  robust  discussion  within  a 
team,  find  specific  lessons  learnt  within  the  organisation,  or  question  a  team  leader  safely 
(Watkins  &Marsick,  2003;  Garvin,  Edmondson,  &Gino,  2008).  Using  multiple  dimensions 
within  a  nominal  definition  for  a  complex  construct  allows  for  a  more  complete 
exploration  of  the  phenomena. 

3.2.3  Operational  Definition 

The  nominal  definition  is  used  as  a  basis  for  generating  the  operational  definition,  and 
therefore,  the  measures  used.  This  cascade  of  definitions  aims  to  generate  transparent, 
specific,  observable  and  justifiable  measures  that  are  readily  traceable  back  to  the 
characterisation  of  the  conceptual  definition.  Watkins  and  Marsick  (2003)  have  included 
measures  of  cognitive,  technical,  social,  and  cultural  dimensions.  Redding  (1997)  reviewed 
several  LO  assessment  measures  available  at  the  time,  and  found  that  the  framework 
created  by  Watkins  and  Marsick  (2003)  was  among  the  few  that  covered  learning  at 
individual,  team  and  organisational  levels.  The  Dimensions  of  Learning  Organisation 
Questionnaire  (DLOQ)  developed  by  Watkins  and  Marsick  (2003)  is  based  on  their  model 
in  which  organisational  learning  occurs  in  these  three  levels  (individual,  team  and 
organisational),  and  covers  the  following  explicit  organisational  characteristics  which  can 
then  be  measured: 

•  Individual  level 

o  Dialogue  and  inquiry,  robust  discussion  and  asking  questions. 

o  Continuous  learning:  opportunities  and  resources  for  learning,  training  and 
education. 

•  Team  level 
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o  Team  learning  and  collaboration:  support  within  teams  for  sharing 
information  and  supporting  each  other's  learning. 

•  Organisational  level 

o  Leadership  supporting  learning:  extent  of  leadership  enables  learning  by 
practice  and  by  resourcing. 

o  Embedded  systems:  information  systems  access  and  utility. 

o  System  connections:  does  the  organisation  support  broader  perspectives. 

o  Shared  vision  and  empowerment:  degree  of  autonomy  and  engagement  with 
the  organisation's  vision  or  mission. 

Goh  and  Richard  (1997)  developed  the  Organizational  Learning  Survey  (OLS)  by 
examining  the  commonalities  within  the  published  literature,  and  using  Garvin's  (1993) 
definition  as  a  guiding  construct,  they  extracted  five  common  dimensions  within  the 
theoretical  and  empirical  literature  as  an  operational  definition.  For  Goh  and  Richard's 
(1997)  a  learning  organisation  would  exhibit  the  following  measurable  characteristics: 

•  Clarity  of  purpose  or  mission:  the  degree  to  which  employees  have  a  clear  vision  and 
understand  how  they  engage  with  it. 

•  Leadership  commitment  and  empowerment:  the  role  of  the  leaders  in  the  organisation 
with  respect  to  helping  employees'  learn  and  elicit  behaviours  that  are  consistent 
with  experimentation  and  changing  culture. 

•  Experimentation  and  rewards":  the  degree  of  freedom  employees  enjoy  in  the  pursuit 
of  new  ways  of  getting  the  job  done  and  freedom  to  take  risks. 

•  Transfer  of  knowledge:  the  systems  that  enable  employees  to  learn  from  others,  from 
past  failures  and  from  other  organisations. 

•  Teamwork  and  group  problem  solving:  the  degree  of  teamwork  possible  in  the 
organisation  to  solve  problems  and  generate  new  and  innovative  ideas. 

The  Learning  Organisational  Survey  (LOS)  is  an  organisational  learning  tool  developed  by 
Garvin,  Edmondson  and  Gino  (2008).  The  LOS  seeks  to  measure  organisational  learning 
and  associated  learning  process.  Learning  factors,  which  when  taken  as  a  whole  represent 
the  elements  of  learning  in  an  organisation  include: 

•  Information-sharing  patterns:  the  extent  to  which  people  share  information. 

•  Inquiry  climate":  the  ways  and  extent  to  which  experimentation,  challenge  and 
inquiry  characterise  the  behaviours  of  organisational  members. 

•  Learning  practices":  refers  to  the  specific  activities  organisational  members  engage  in 
to  learn. 

•  Achievement  mindset:  focus  on  results  and  task  outcomes. 

The  Learning  Environment  Questionnaire  (LEQ)  developed  by  Armstrong  and  Foley 
(2003)  is  an  Australian  questionnaire  that  focuses  on  the  structural  level  mechanisms 
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indicative  of  learning  organisations  rather  than  any  individual  characteristics  that  effect 
learning.  The  LEQ  was  not  intended  to  identify  any  individual  learning  processes,  rather, 
it  aimed  to  map  the  underlying  organisational  culture,  processes  and  structures  that  create 
and  improve  learning  opportunities.  Consequently,  the  LEQ  has  a  different  focus  to  the 
other  learning  organisation  questionnaires.  Such  cultural  and  structural  factors  do  indeed 
provide  the  context  within  which  learning  is  situated  and,  according  to  Armstrong  and 
Foley  (2003),  are  the  organisational  learning  mechanisms  (OLM).  The  four  categories  of 
OLM  identified  by  Armstrong  and  Foley  (2003)  were: 

•  learning  environment 

•  identifying  learning  and  development  needs 

•  meeting  learning  and  development  needs 

•  applying  learning  to  the  workplace. 

The  LEQ  has  12  scales  to  measure  the  four  categories,  which  have  been  developed  to 
measure  the  "cultural  and  structural  facets  of  a  learning  organisation"  (Armstrong  &Foley, 
2003,  P80). 

The  instruments  described  in  the  Table  1  have  all  been  based  on  the  theoretical  and 
empirical  understanding  of  what  constitutes  a  learning  organisation.  Each  instrument  has 
specific  items  associated  with  each  measure.  As  an  exemplar,  Marsick  and  Watkins'  (2003) 
DLOQ,  the  dimension  'dialogue  and  inquiry'  asks  to  what  extent  does  your  team: 

•  "Give  open  and  honest  feedback" 

•  "Listen  to  other's  views  before  speaking" 

•  "Are  encouraged  to  ask  'why?'  regardless  of  rank?" 

The  instruments  all  have  a  coherent  theoretical  link  from  the  conceptual  definition  of  the 
learning  organisation  to  the  operational  measures  provided  within  the  measures.  This 
means  that  the  instruments  do  indeed  provide  acceptable  levels  of  theoretical  validity. 


3,3  Reliability  of  Learning  Organisation  Measures 

The  items  that  are  used  within  each  of  these  instruments  were  found  to  possess  an 
acceptable  level  of  theoretical  validity  within  their  target  populations.  The  measures 
focused  on  observable,  specific  and  justifiable  concrete  behaviours.  Their  reliability  was 
then  evaluated  using  a  variety  of  criteria,  as  mentioned  in  Section  2.1.1.  The  DLOQ,  OLS 
and  LEQ  all  have  published  statistical  reliability,  and  demonstrated  acceptable  levels  (that 
is,  Cronbach  alphas  above  .70). 

In  addition,  the  DLOQ  and  OLS  have  reported  their  discriminant  validity  criteria, 
demonstrating  an  acceptable  level  of  discrimination  between  different  populations.  Goh 
and  Richards  (1997)  reported  on  the  ability  of  the  OLS  to  discriminate  between  four  very 
different  organisations:  (i)  a  government  public-sector  organisation;  (ii)  a  high  performing 
high-tech  company;  (iii)  a  'qago'  (a  quasi-autonomous  government  organisation);  and  (iv) 
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a  large  telecommunication  company  (private  sector).  The  OLS  differentiated  between  the 
work  practices  of  the  four  organisations,  with  all  four  organisations  varying  by  the 
hypothesised  direction  given  the  theoretical  understanding  of  organisational  structure, 
performance  and  learning  (with  the  government  organisation  the  lowest  scoring  and  the 
high-performing  high-tech  company  having  the  highest  scores).  Also,  the  OLS  was 
correlated  significantly  with  job  satisfaction  across  all  organisations  (r=0.64,  p<.001), 
showing  that  supportive  learning  environments  were  linked  to  increased  job  satisfaction. 

Marsick  and  Watkins'  (2003)  DLOQ  has  an  acceptable  level  of  reliability  as  reported  by 
Yang  (2003)  and  Yang,  Marsick  and  Watkins,  (2004).  The  reliability  (or  internal 
consistency)  scores  for  Cronbach's  coefficient  alphas  were  0.80  to  0.87  with  an  overall 
reliability  of  0.96  (Yang,  2003:160).  Their  questionnaire  comprises  55  statements  concerning 
organisational  practices  which  relate  to  learning  in  and  by  organisations.  Respondents  are 
invited  to  indicate  the  extent  to  which  they  perceive  these  practices  occurring  within  their 
organisation  by  way  of  a  six  point  Likert  scale. 

Armstrong  and  Foley's  (2003)  LEQ  reported  acceptable  reliability  levels  (Cronbach's  above 
0.7  for  all  the  dimensions)  and  also  strong  theoretical  construct  validity,  however  no  other 
validity  criteria  were  satisfied.  The  focus  of  the  LEQ  was  different  to  the  other  Learning 
Organisation  questionnaires;  the  LEQ  is  solely  aimed  at  measuring  the  structural  and 
cultural  environment  that  supports  learning  within  an  organisation.  That  the  LEQ  focuses 
on  structural  and  cultural  level  mechanisms  is  not  a  fault;  however,  given  that  the  aim  of 
the  Army  Learning  Organisation  Questionnaire  is  to  measure  learning  at  the  individual 
and  team  levels  as  well  as  at  the  organisational  (structural  and  cultural)  level,  the  LEQ  was 
deemed  not  to  be  sufficient. 

The  Organisational  Learning  Survey  (OLS)  developed  by  Garvin,  Edmondson  and  Gino 
(2008)  examines  the  extent  to  which  an  organisation  exhibits  the  characteristics  and 
behaviours  associated  with  the  following  building  blocks  of  a  learning  organisation: 
supportive  learning  environment;  concrete  learning  processes  and  practices;  and 
leadership  behaviours  that  reinforce  learning.  These  building  blocks  represent  the  three 
main  sections  of  the  on-line  diagnostic  survey  instrument.  Unfortunately,  Garvin, 
Edmondson  and  Gino's  (2008)  survey  instrument  is  not  publically  available,  as  it  is  only 
available  online  through  the  author's  own  website.  Nor  did  Garvin  et  al  (2008)  report  any 
reliability  or  validity  data  against  which  to  evaluate  their  questionnaire. 


UNCLASSIFIED 


13 


DSTO-TN-1325 


UNCLASSIFIED 


Table  2:  Learning  Organisation  questionnaires:  validity  and  reliability  criteria 


Questionnaires 

Scale  -  Reliability 

Scale  -  Validity 

Internal  consistency 

Inter-rater 
reliability  and 
agreement 

Content  validity 

Discriminate 

validity 

Statistically 
validated 
relationships  to 
outcomes  of  interest 

Dimensions  of 

Learning 

Organization 

Questionnaire 

(DLOQ) 

Yes  -  reported 
overall  coefficient 
alpha  0.97 

None  reported 

Yes  -  Structural 
Equation  Model 
(Yang,  2003)  of  7 
dimensions  for 

DLOQ  best  fit. 

Yes  -  reported  in 
Yang  (2003). 

Yes,  results  of 

SEM  reported  in 
Yang  (2003). 

Organizational 

Learning 

Survey  (OLS) 

Yes  -  overall 
Cronbach 
reported  0.90 

Yes  -  test/re- 
test  correlation 
0.77 

Yes  - 

normative/theoreti 
cal  approach 

Yes  - 

discriminated 
between  differing 
organisations 

Yes,  significantly 
correlated  to: 
job  satisfaction, 
job  formalisation, 
profitability 

Learning 

Environment 

Questionnaire 

(LEQ) 

Yes  -  Cronbachs 
for  scales  reported 
between  0.72  and 
0.92 

Yes  -  varied 
between  0.22 
to  0.86 

Yes  -  Theoretical 
structural  and 
cultural  relationship 

None  reported 

None  reported 

Learning 
Organization 
Survey  (LOS) 

None  reported 

None  reported 

Yes  -based  on 
Garvin's  theoretical 
work 

None  reported 

None  reported 

4.  Results:  Developing  the  Army  Learning  Organisation 

Questionnaire 

Using  the  established  method  of  scale  development  as  described  by  Streiner  and  Norman 
(2003,  2008)  the  literature  was  reviewed  for  established  measures.  The  measures  were 
critically  reviewed  against  specific  reliability  and  validity  criteria.  All  four  measures  were 
found  to  have  a  strong  theoretical  basis;  all  four  were  judged  to  have  an  acceptable  level  of 
content  validity.  The  LEQ  was  found  to  not  cover  all  the  necessary  organisational 
dimensions  required  for  the  ALOQ.  Applying  further  criteria,  acceptable  information  of 
empirical  validity  was  reported  for  only  two  of  the  four  instruments,  with  at  least  one 
criteria  of  validity  (either  discriminant  or  Structural  Equation  Modelling  [SEM]  or 
statistical  construct  validity).  These  two  measures  were  Marsick  and  Watkins'  (2003) 
DLOQ  and  Goh  and  Richard's  (1997)  OLS.  The  other  two  measures,  LEQ  and  the  LOS  did 
not  have  any  other  empirical  validation  processes  reported. 

Of  the  four  instruments,  the  DLOQ,  OLS  and  the  LEQ  did  report  reliability  outcomes  for 
target  populations.  Of  these  three  instruments,  the  DLOQ  and  OLS  were  found  to  be 
above  the  acceptable  level  of  0.70  (Nunnally,  1978).  Thus,  as  the  DLOQ  and  OLS  were 
found  to  be  reliable  and  valid  instruments  for  measuring  Learning  Organisation 
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characteristics,  they  were  selected  to  be  adapted  to  the  Army  context.  The  two 
questionnaires'  were  amalgamated,  their  language  and  question  construction  were 
adapted  to  military  and  the  resulting  questionnaire  was  termed  the  Army  Learning 
Organisation  Questionnaire  (ALOQ). 


4.1  ALOQ  Pre-test  and  Testing  Process 

Following  Streiner  and  Norman's  (2003;  2008)  method  for  developing  questionnaires,  the 
items  within  the  DLOQ  and  OLS  were  then  piloted  with  the  target  population  in  order  to 
ensure  item  (or  question)  clarity  and  comprehension.  The  questionnaire  went  through  a 
three  phase  iterative  pre-test  process  to  ensure  that  the  language  was  comprehensible, 
unambiguous  and  appropriate  for  the  Army. 

4.1.1  First  Pretest;  In-house  DSTO  Army  Personnel 

The  initial  pilot  testing  was  conducted  with  in-house  military  personnel  in  DSTO.  The 
military  personnel  completed  the  questionnaire,  then  provided  their  feedback  on  the 
language  and  the  applicability  of  the  types  of  questions  to  the  Australian  Army.  The 
feedback  was  conducted  through  face-to-face  interviews.  The  personnel  were  asked  for 
their  interpretation  of  the  questions,  and  what  they  thought  about  when  they  were 
answering  particular  questions.  The  first  pilot  phase  was  conducted  on  eleven  personnel. 
The  military  personnel  were  all  either  senior  non-commissioned  officers  (WOl,  W02)  or 
commissioned  officers. 

The  feedback  was  collated,  and  the  items  were  adapted  based  on  this  feedback.  Most  of  the 
feedback  pertained  to  the  following  areas: 

•  concerns/ difficulties  in  answering  "organisational  performance"  scales 

•  concerns  about  "corporate  speak"  within  the  questionnaire 

•  concerns  about  some  of  the  item  constructs  including  double-barrelled  item 
construction2 

•  clarifying  answer  rating  scales  to  match  the  items. 

4.1.2  Second  Pre-test:  Land  Warfare  Development  Centre  Workshop  -  Army 
Personnel 

The  second  pilot  was  conducted  on  30  military  personnel  attending  a  workshop  with 
ranks  ranging  from  CPL  to  LTCOL.  The  questionnaire  was  distributed  and  completed 
prior  to  the  workshop,  and  feedback  provided  during  a  workshop  session. 

The  feedback  was  collated  and,  again,  the  questionnaire  was  amended  in  response  to  these 
concerns.  The  main  issues  were: 


2  An  example  of  a  double  barrelled  item;  "I  can  talk  to  my  supervisor  and  team-mates  openly  and 
honesty". 
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•  concerns/ difficulties  in  answering  "organisational  performance"  scales 

•  concerns  rose  about  whether  less  educated  or  experienced  personnel  would 
understand  some  of  the  items. 

4.1.3  Third  Pre-test:  Soldier  Interviews 

The  final  phase  of  the  piloting  process  aimed  to  address  the  concerns  about  how  the 
"diggers"  ("other  ranks"  or  private  soldiers)  would  interpret  the  items.  Twelve  face-to-face 
interviews  were  conducted,  which  were  audio  recorded,  during  which  soldiers  completed 
the  questionnaire  and  were  then  asked  what  they  believed  the  questionnaire  was  asking 
and  how  they  answered  it.  For  example,  interviewees  were  asked  "what  did  you  think 
about  when  we  asked  about  empowerment?"  or  "what  did  you  understand  when  you 
answered  Question  Four?"  Their  responses  were  collated,  and  common  themes  were 
distilled.  For  example,  "when  I  was  asked  about  'empowerment/  I  was  thinking  if  I  could 
do  what  I  needed  to  do  to  get  a  job  done.  Was  I  allowed  to  do  things  that  I  needed  to  do?" 

This  provided  a  test  of  construct  validity,  showing  that  when  soldiers  were  asked  items 
that  had  been  identified  by  the  previous  pilot  phases  as  being  potentially  difficult  to 
understand,  that,  in  fact,  they  did  comprehend  what  was  being  asked  of  them. 

Because  both  Phase  1  and  Phase  2  personnel  found  it  very  difficult  to  answer  questions  on 
organisational  performance  measures  these  items  were  eliminated  from  the  questionnaire 
including  knowledge  performance  and  organisational  performance  dimensions.  For  example 
"knowledge  performance"  included  items  such  as  "in  my  unit  or  workgroup,  the  number 
of  personnel  learning  new  skills  is  greater  than  last  year"  (answering  from  "strongly 
disagree"  to  "strongly  agree")  while  organisational  performance  included  the  items  "in  my 
unit  or  workgroup,  we  completed  more  training  this  year  than  last  year"  (answering  from 
"strongly  disagree"  to  "strongly  agree"). 

4.2  ALOQ  Reliability  and  Validity 

This  section  reports  the  ALOQ's  reliability  and  validity  using  the  criteria  discussed  in 
Section  2.1.1  (Table  2).  The  ALOQ  is  made  up  of  the  DLOQ  and  OLS,  and  in  doing  so,  it 
incorporates  the  previous  work's  published  reliability  and  validity  measures.  The  ALOQ 
was  administered  to  an  Army  brigade  (N=390).  The  Cronbach  alphas  were  then  calculated 
from  this  sample.  The  exploratory  factor  analysis  (EFA)  was  then  conducted  on  a  larger 
sample  of  Army  personnel  (N=3700)  in  order  to  evaluate  statistical  validity. 

4.2.1  ALOQ  Reliability 

4. 2.1.1  ALOQ  Internal  Consistency 

The  internal  consistency  of  the  ALOQ  dimensions  is  reported  in  Table  3:  ALOQ  Cronbach 
alpha  and  all,  except  one,  was  above  the  recommended  0.70  level.  The  measure 
"innovation  and  experimentation"  showed  an  internal  consistency  of  0.68  however,  with 
the  subsequent  dropping  of  the  item  with  the  lowest  item-scale  correlation  coefficient  (i.e. 
Question  60:  "I  often  bring  new  ideas  to  my  workgroup  or  section")  the  measure 
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subsequently  incorporated  four  items  with  a  Cronbach  alpha  of  0.72.  Overall,  the  ALOQ 
demonstrated  acceptable  level  of  statistical  reliability. 

Table  3:  ALOQ  Cronbach  alpha 


ALOQ  Measure 

Modified 

from 

Cronbach 

alpha 

1 

Continuous  learning 

DLOQ 

0.81 

2 

Dialogue  and  inquiry 

DLOQ 

0.84 

3 

Team  work  and  collaboration 

DLOQ 

0.83 

4 

Systems  -  capture  and  share  learning 

DLOQ 

0.84 

5 

Collective  vision 

DLOQ 

0.88 

6 

Connection  -  organisation  to  environment 

DLOQ 

0.86 

7 

Strategic  Leadership  for  learning 

DLOQ 

0.90 

8 

Leadership  -  empowerment 

OLS 

0.73 

9 

Innovation  and  experimentation 

OLS 

0.68 

10 

Knowledge  transfer 

OLS 

0.79 

11 

Teamwork-  diversity 

OLS 

0.78 

4.2.2  ALOQ  Validity 

4.2.2.1  ALOQ  Content  and  Face  Validity 

Both  the  DLOQ  and  OLS  were  generated  from  within  a  theoretical  conception  of  a 
Learning  Organisation  using  the  following  process: 

1.  Conceptualisation:  What  is  an  ideal  learning  organisation?  One  that  responds 
successfully  to  a  changing  environment. 

2.  Nominal  definition:  what  would  it  look  like?  What  makes  it  different  from  a  non¬ 
learning  organisation?  For  example,  the  organisation  would  have  better  quantity 
and  quality  of  communication. 

3.  Operational  definition:  what  exactly  do  we  mean  by  "better  communication  is 
important  for  learning  in  organisations"?  For  example,  personnel  can  discuss  their 
own  and  other's  mistakes  with  teams  or  supervisors  when  they  need  to. 

4.  Measures:  the  measure  then  became  "to  what  extent  can  you  talk  about  mistakes 
with  your  supervisor/ s?  To  you  teammates?" 

Thus  the  ALOQ's  content  validity  is  firmly  anchored  within  the  theoretical  literature, 
building  on  the  original  authors'  content  validation. 

The  benefit  of  the  ALOQ's  face  validity  was  judged  to  outweigh  the  potential  issue  of 
social  desirability  bias.  Face  validity  provides  respondents  with  relevant  questions  and 
helps  provide  feedback  to  the  main  stakeholders.  "Faking  good"  (or  social  desirability)  is 
always  an  issue  when  items  have  face  validity  and,  hence,  should  be  minimised  wherever 
possible  by  anonymous  self-administration  of  questionnaires,  with  anonymous  and 
confidential  collection  of  questionnaires  (Streiner  &  Norman,  2003;  2008).  Thus,  within  the 
ALOQ,  the  potential  social  desirability  bias  was  moderated  by  using  anonymous 
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administration  techniques;  for  example,  by  confidential  ballot  box  returns  and  paper-pen 
responses. 


4.3  ALOQ  Statistical  Analysis 

The  next  step  in  investigating  ALOQ  validity  was  to  conduct  statistical  analyses  using 
exploratory  factor  analysis  (EFA)  in  order  to  empirically  investigate  the  applicability  of  the 
original  authors'  model  of  learning  organisation  to  Australian  Army  data. 

4.3.1  Participants 

The  data  set  was  collected  from  3,895  respondents  of  the  ALOQ.  This  included  data  from 
3275  Regular  Army  and  552  Reservists.  Respondents  were  deleted  list-wise3  if  they  left 
greater  than  1/3  of  the  items  unanswered.  Valid  respondents  represented  96%  of  all 
respondents  -  a  loss  of  4%  of  respondents. 

Of  the  3,724  valid  respondents,  91%  of  the  sample  was  male,  and  38%  were  between  18 
and  25  years  old.  The  majority  (66%)  of  personnel  were  ORs  (other  ranks),  17%  were  NCOs 
and  15%  were  commissioned  officers. 

4.3.2  Treatment  of  missing  data 

As  noted  above,  103  cases  were  deleted  list-wise  due  to  missing  data.  Of  the  remaining 
3,724  valid  respondents  there  was  a  small  proportion  of  unanswered  questions.  A  Full 
Information  Maximum  Likelihood  (FIML)4  method  was  used  to  estimate  replacement 
values  for  the  missing  data. 

4.3.3  Exploratory  factor  analysis  (EFA)  of  ALOQ 

The  total  of  44  items  (made  up  of  43  items  developed  by  Mar  sick  and  Watkins  (2003)  with 
one  additional  Army-specific  item)  together  with  the  21  items  developed  by  Goh  and 
Richards  (1997)  were  used  in  the  EFA.  Both  item  sets  (44  items  and  21  items)  were 
theoretically  measuring  eleven  underlying  constructs  (see  Appendix  A).  Exploratory  factor 
analysis  (EFA)  was  conducted  on  the  two  datasets  in  order  to  evaluate  the  two  different 
theoretical  constructs,  both  Yang,  Marsick  and  Watkins'  (2004)  and  Goh  and  Richard's 
(1997)  models.  The  aim  was  to  investigate  to  what  extent  Army's  data  aligned  with  these 
specific  theoretical  models  in  order  to  assess  their  validity;  consequently  the  two  data  sets 
were  analysed  separately.  There  is  a  large  theoretical  overlap  between  both  models'  and 
this  interaction  is  planned  to  be  investigated  further  in  further  reports. 


3  List-wise  deletion  is  a  method  for  handling  missing  data;  in  this  case,  if  l/3rd  or  more  of  a 
respondent's  data  was  missing,  then  their  entire  record  was  excluded  from  the  analysis. 

4  FIML  Full  Information  Maximum  Likelihood  estimation  method  utilises  all  available  information 
from  the  data  when  there  is  a  high  proportion  of  observations  with  random  missing  values. 
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EFA  considers  the  correlations  between  all  of  the  measures  under  consideration  and  asks 
the  question,  "Are  there  subsets  of  measures  where,  for  each  subset,  the  measures  are 
highly  correlated  with  each  other,  but  are  not  correlated  with  any  of  the  other  measures?" 
If  such  subsets  are  found,  it  is  assumed  they  are  measuring  the  same  common  underlying 
construct.  The  following  sections  describe  this  analysis. 

4.3. 3.1  Checking  correlations  and  commonalities 

The  first  step  in  the  EFA  was  to  determine  to  what  extent  the  items  (or  variables)  are 
correlated  to  each  other.  The  adequacy  of  the  correlations'  magnitude  was  initially  tested 
by  using  Kaiser-Meyer-Oklin  (KMO)  (Kaiser,  1974)  and  Bartlett's  test  of  sphericity  (see 
Table  4). 


Table  4:  EFA  assumption  testing  ofALOQ:  Adequacy  of  correlation  strength 


Assumption  tests 

44  item  set* 

21  item  set 

KMO  measure  of  sampling  adequacy 

0.97 

0.94 

Bar  left's  test  of  sphericity 

Approx  chi-square 

98210.326 

25899.448 

DF 

946 

136 

Significance 

P  <.0001 

P  <.0001 

*  The  44  item  set  was  made  up  of  Marsick  and  Watkin's  (1996)  43  items  plus  one  Army  specific 
item. 


The  KMO  statistic  measures  whether  the  correlations  between  pairs  of  variables  can  be 
explained  by  other  variables  -  a  necessary  condition  to  support  the  existence  of  an 
underlying  factor  structure.  Kaiser  (1974)  described  KMO  factors  greater  than  0.9  as 
marvellous,  and  values  less  than  0.5  as  unacceptable.  The  KMO  for  DLOQ  and  OLS  item  sets 
are  over  0.9  indicating  that  the  pairs  of  variables  are  strongly  explained  by  other  variables; 
there  is  a  factor  structure  to  the  data. 

Bartlett's  test  of  sphericity  indicates  to  what  extent  the  correlation  matrix  differs 
significantly  from  an  identity  matrix;  are  the  relationships  between  the  variables  amenable 
to  modelling  using  Structural  Equation  Modelling  (SEM)?  A  very  large  Chi-square  for 
both  DLOQ  and  OLS  items  with  p  values  of  less  than  .001  indicates  that  the  data  does 
indeed  differ  significantly  from  an  identity  matrix  and,  thus,  there  are  significant 
correlations  overall  between  the  variables. 

Another  means  of  assessing  the  strength  of  the  linear  association  is  to  look  at  the  variance 
explained  in  each  variable  by  either  all  variables  or  by  the  extracted  factors.  A  squared 
multiple  correlation  coefficient  (R2)  is  a  measure  of  this  variance  and  is  called  a 
communality  in  EFA.  An  initial  communality  is  the  variance  explained  in  a  single  variable 
by  all  other  variables;  extraction  communality  is  the  variance  explained  in  a  single  variable 
by  the  extracted  factors.  A  low  communality  (usually  below  0.3)  suggests  that  the  variable 
may  have  little  to  do  with  the  other  variables  and  therefore  may  be  dropped  from  the 
analysis.  The  communalities  for  the  ALOQ  are  shown  in  Table  10:  ALOQ  communalities  in 
Appendix  A.  The  communalities  show  that  the  variation  within  the  data  can  be  explained 
by  underlying  constructs. 
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4.3. 3. 2  ALOQ  Factor  extraction,  Eigen  Values,  Scree  Plots  and  the  Factor  Matrix 
A  number  of  factors  were  extracted  to  adequately  describe  the  measures.  Choosing  the 
most  appropriate  extraction  method  and  determining  the  correct  number  of  factors  needs 
some  consideration.  To  evaluate  the  options  available,  it  is  first  necessary  to  understand 
the  aim  of  this  activity.  The  differences  amongst  the  extraction  methods  published  can  be 
explained  by  their  objectives  (Pallant,  2007;  Holmes-Smith,  2012);  for  example.  Principle 
component  (PC)  extraction  aims  to  generate  linear  combinations  of  variables  so  that  the 
variance  of  the  first  factor  and  each  subsequent  orthogonal  combination  is  maximised.  In 
Maximum  likelihood  (ML)  extraction,  the  objective  is  to  find  the  linear  combination  of 
variables  such  that  when  the  first,  and  subsequent  orthogonal  combination,  are  used  to 
reproduce  the  sample  correlation  matrix,  the  difference  between  the  reproduced 
correlation  matrix  and  the  observed  sample  correlation  matrix  is  minimised. 

Fabrigar  et  al.  (1999)  recommend  the  use  of  Maximum  Likelihood  (ML)  extraction  if  there 
are  no  concerns  with  data  distribution,  for  example,  no  significant  skewness  and  kurtosis. 
Inspecting  the  44  items  showed  that  the  data  were  only  mildly  skewed  (values  ranging 
from  -0.250  to  0.480)  and  kurtosis  was  mild  (values  range  from  -0.554  to  0.127)  so  ML 
extraction  was  used.  Regardless  of  the  extraction  method,  if  all  the  measures  are  expressed 
in  a  standardised  form,  for  example,  a  mean  of  zero  with  a  standard  deviation  of  one,  then 
the  total  variance  will  equal  the  number  of  measures.  It  then  becomes  possible  to  calculate 
how  much  of  the  total  variance  is  accounted  for  by  each  factor  (Table  11:  Variance 
explained  (Eigen  values)  for  44  item  data  set).  The  first  factor  explains  a  total  of  43%  of  the 
total  variance  (18.9  of  a  total  variance  of  44).  The  second  has  an  Eigen  value  of  2.1  (of  a 
total  of  44)  and  explains  4.8%  of  the  total  variance. 
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There  are  two  main  methods  for  selecting  the  number  of  factors;  first,  to  select  all  those 
factors  with  an  Eigen  value  greater  than  one  and  is  known  as  Kasier's  criterion  (Pallant, 
2007)  and  the  second  uses  Catell's  scree  test  (Cattell,  1966),  which  uses  a  plots  the  Eigen 
value  against  the  number  of  factors  in  determining  the  cut  off  factor  (Cattell,  1966).  Using 
Catell's  scree  test  counters  the  criticisms  of  using  Kraiser's  criteria  (Pallent,  2007).  Kraiser's 
criterion  has  been  criticised  for  a  lack  of  stringency;  that  is,  retaining  too  many  factors. 
Catell's  scree  test  provides  a  more  informed  selection.  Using  the  scree  plot  method 
recommended  by  Cattell  (1966)  a  nine  factor  solution  was  initially  forced5  onto  this 
analysis  for  the  44  item  data  set.  Upon  inspection  of  the  results,  the  ninth  factor  was  found 
to  be  a  single  item  so  the  factors  were  reduced  to  eight  (Table  5).  A  single  item  factor  is  not 
suitable  for  further  analysis.  For  the  21  item  data  set,  the  scree  plot  method  also  found  that 
six  factors  best  described  the  data;  hence  a  six  factor  solution  was  initially  forced  onto  the 
analysis.  The  factors  were  then  rotated  allowing  for  easier  interpretation.  There  are  a 
number  of  different  rotation  methods  available;  an  oblique  rotation  should  be  used  if  there 
is  a  theoretical  reason  for  the  factors  to  be  correlated.  The  factors  have  been  shown  to  be 
correlated  (KMO  statistic  was  over  0.90)  so  a  Promax  Rotation6  was  performed.  SPSS 
produces  a  separate  pattern  matrix  (containing  the  factor  loadings  showing  how  much 
weight  are  assigned  to  each  factor)  and  a  structure  matrix  (containing  the  correlations 
between  the  factors  and  the  variables).  The  pattern  matrix  of  the  44  items  loading  onto 
eight  factors  is  shown  in  Table  5  while  the  pattern  matrix  for  the  21  item  data  set  is  shown 
in  Table  6. 


Table  5-  ALOQ  44  item  pattern  matrix  showing  eight  factor  loading  solution 


Factor  j 

i 

2 

3 

4 

5 

6 

7 

8 

Ql 

0.811 

Q2 

0.807 

Q3 

0.498 

Q4 

0.377 

Q5 

0.472 

Q6 

0.471 

Q7 

0.726 

Q8 

0.386 

0.366 

Q9 

0.644 

Q10 

0.386 

Qll 

0.562 

Q12 

0.869 

Q13 

0.660 

Q14 

Q15 

0.633 

Q16 

0.743 

Q17 

0.616 

Q18 

0.360 

0.514 

Q19 

0.519 

0.343 

Q20 

0.469 

Factor  j 

5  "Forcing"  is  the  technical  term  for  specifying  the  number  of  factors  desired  within  the  factor 
analysis  solution. 

6  Promax  rotation  is  a  non-orthogonal  (oblique)  rotation  method  which  is  computationally  faster 
than  the  direct  oblimin  method  and  thus  is  more  useful  for  large  data  sets. 
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1 

2 

3 

4 

5 

6 

7 

8 

Q21 

0.309 

0.382 

Q22 

0.721 

Q23 

0.561 

Q24 

0.409 

Q25 

0.344 

Q26 

0.565 

Q27 

0.836 

Q28 

0.840 

Q29 

0.739 

Q30 

0.889 

Q31 

0.637 

Q32 

Q33 

0.436 

Q34 

0.684 

Q35 

0.516 

Q36 

0.648 

Q37 

0.536 

Q38 

0.373 

Q39 

0.520 

Q40 

0.757 

Q41 

0.633 

Q42 

0.932 

Q43 

0.855 

Q44 

0.609 

Table  6  -  ALOQ  21  items  pattern  matrix  showing  six  factor  loading  solution 


Factor  | 

i 

2 

3 

4* 

5 

6 

Q45 

0.691* 

Q46 

0.510 

Q47 

0.623 

Q48 

0.932 

Q59 

0.589 

Q49 

0.587 

0.302 

Q50 

0.429 

Q51 

0.669* 

Q60 

0.534 

Q61 

0.708 

Q52 

0.717 

Q53 

0.628 

Q54 

0.700 

Q55 

0.846 

Q56 

0.575 

Q57 

0.490 

Q58 

0.343 

*Factor  is  a  method  effect  due  to  negatively  worded  items  therefore  this  factor  is  discounted  from 
further  analyses. 


The  exploratory  factor  analysis  of  the  ALOQ  found  underlying  factors  that  emerged  from 
our  target  respondents  that  instead  of  the  Marsick  and  Watkins'  (2003)  seven  factors,  we 
found  that  there  were  eight  factors  emerging  from  our  data.  In  line  with  the  Goh  and 
Richard's  (1997)  original  items,  four  items  were  also  found  within  the  EFA. 
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The  ALOQ,  based  on  the  results  of  the  EFA,  generally  aligns  well  with  the  established 
evidence  of  measuring  learning  within  organisations.  However  one  difference  was  found 
within  our  data  compared  to  DLOQ;  the  continuous  learning  practices  dimension  was 
separated  into  two,  resources  and  reivards  and  learning  practices.  This  result  may  well  reflect 
the  differences  within  the  Army's  organisational  structure  and  processes  compared  to  the 
organisations  considered  during  DLOQ  development;  for  example,  that  rewards  and 
resources  are  seen  as  being  controlled  centrally  in  Army  and  thus  are  not  reflective  of 
individual's  perceptions  of  their  own  teams  or  particular  work  environments.  More 
detailed  results  from  the  EFA  are  presented  in  Appendix  A.3. 

Six  factors  were  initially  found  within  the  data,  in  comparison  to  Goh  and  Richard's  (1997) 
originally  hypothesised  five  factors.  Inspecting  the  pattern  matrix  showed  that  Q45  and 
Q51  (the  two  negatively  worded  items)  were  in  fact  loading  onto  one  factor  (Factor  4);  this 
indicates  that  this  loading  is  a  method  effect  (the  detected  co-variation  due  to  being 
negative  phrased).  Consequently,  this  factor  (Factor  4)  was  discounted.  The  exploratory 
factor  analysis  of  the  ALOQ  found  that  were  five  underlying  factors  that  emerged  from 
our  target  respondents  in  line  Goh  and  Richard's  (1997)  five  factor  model  (Table  7). 


Table  7:  ALOQ  EFA  emergent  factors 


ALOQ  Measure 

From 

ALOQ  EFA  factor 

1 

Continuous  learning 

DLOQ 

1.1 

Learning  practices 

1.2 

Resources  and  rewards 

2 

Dialogue  and  inquiry 

DLOQ 

2.1 

Dialogue  and  inquiry 

3 

Team  work  and  collaboration 

DLOQ 

3.1 

Teamwork 

4 

Systems  -  capture  and  share  learning 

DLOQ 

4.1 

Systems  to  capture  and  share  learning 

5 

Collective  vision 

DLOQ 

5.1 

Collective  vision 

6 

Connection  -  organisation  to  environment 

DLOQ 

6.1 

Connection  -  organisation  to  environment 

7 

Strategic  leadership  for  learning 

DLOQ 

7.1 

Strategic  leadership  for  learning 

8 

Leadership  -  empowerment 

OLS 

8.1 

Leadership  -  empowerment 

9 

Innovation  and  experimentation 

OLS 

9.1 

Innovation  and  experimentation 

10 

Knowledge  transfer 

OLS 

10.1 

Knowledge  transfer 

11 

Teamwork  and  diversity 

OLS 

11.1 

Teamwork  and  diversity 

The  EFA  demonstrates  that  the  underlying  commonality  within  the  ALOQ  items  are 
determined  by  the  twelve  latent  factors  in  Table  7,  which  align  with  the  theoretical 
constructs  found  within  the  literature. 


4.4  Summary  of  ALOQ  reliability  and  validity 

The  theoretical  and  statistical  analysis  of  the  ALOQ  demonstrates  that  it  is  a  valid  and 
reliable  measure  of  learning  organisation  characteristics.  The  ALOQ  conforms  to  the 
acceptable  standards  within  the  psychometric  literature  (Table  8).  The  ALOQ  has 
acceptable  levels  of  internal  consistency  (Cronbach's  alpha)  yet  not  excessive  levels  (above 
0.90)  that  may  indicate  overly  correlated  factors  (Streiner  &  Norman,  2003;  2008).  Also,  the 
results  from  the  EFA  indicate  that  the  hypothesised  factors  within  Marsick  and  Watkins 
(2003)  and  Goh  and  Richards  (1997)  are  supported  within  our  data  set,  with  a  slight 
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variation  for  one  DLOQ  factor  loading  into  two.  The  underlying  factors  are  consistent  with 
published  literature  on  organisational  learning  factors  and  thus  support  the  construct 
validation  of  the  ALOQ  with  its  target  respondents. 


Table  8:  ALOQ  validity  and  reliability 


ALOQ  Measure 

Modified 

from 

ALOQ 

Internal 

reliability 

(Cronbach 

alpha) 

ALOQ- 

Test-retest 

reliability 

ALOQ 

Content 

validity 

ALOQ- 

Discriminant 

validity 

ALOQ- 
Validated 
relationships 
to  outcomes  of 
interest 

ALOQ- 
EFA  results 

1 

Continuous 

learning 

DLOQ 

0.81 

DLOQ 

None 

reported 

Yes- 

developed 

from 

theoretical 

basis; 

cognitive 

interviews 

during  pilot 

process 

DLOQ  - 

discriminated 

between 

different 

organisational 

types 

DLOQ; 
correlated  to 
financial 
performance 
and 

knowledge 

performance 

of 

organisations. 

2  factors: 
Learning 
practice, 
rewards  and 

resources 

2 

Dialogue  and 
inquiry 

DLOQ 

0.84 

Single  factor 

3 

Team  work  and 
collaboration 

DLOQ 

0.83 

Single  factor 

4 

Systems  - 
capture  and 
share  learning 

DLOQ 

0.84 

Single  factor 

5 

Collective  vision 

DLOQ 

0.88 

Single  factor 

6 

Connection  - 
organisation  to 
environment 

DLOQ 

0.86 

Single  factor 

7 

Strategic 
leadership  for 
learning 

DLOQ 

0.90 

Single  factor 

8 

Leadership  and 
empowerment 

OLS 

0.73 

OLS; 

correlation 
of  0.77 

OLS; 

discriminated 

between 

differing 

organisations 

OLS; 

correlated  to 
organisational 
commitment, 
job  satisfaction 

Single  factor 

9 

Innovation  and 
experimentation 

OLS 

0.72* 

Single  factor 

10 

Knowledge 

transfer 

OLS 

0.79 

Single  factor 

11 

Teamwork  and 
diversity 

OLS 

0.78 

Single  factor 

*Note  that  one  item  was  removed  from  original  measure,  which  lead  to  an  increase  from  0.68. 


The  ALOQ  can  be  used  confidently  to  measure  the  characteristics  of  learning 
organisations,  especially  of  military  organisations.  Given  the  rigorous  grounding, 
pretesting  and  statistical  analysis  of  the  ALOQ,  valid  conclusions  can  be  drawn  about 
what  exactly  the  questionnaire  is  measuring;  including  the  underlying  factors  such  as  team 
communication,  leadership,  knowledge  transfer  and  generation,  and  learning  practices. 
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5.  Conclusions 


The  existing  instruments  (DLOQ,  OLS,  LOS  and  LEQ)  all  provided  theoretically  based 
measures  of  the  characteristics  of  learning  organisations.  The  measures  were  found  to  be 
both  valid  and  statistically  reliable.  Thus,  the  DLOQ  and  OLS  have  provided  a  useful  basis 
upon  which  to  build  the  ALOQ.  The  available  measures  were  adapted  for  use  within  the 
Army,  such  as  changing  the  items'  wording  using  an  iterative  test-retest  process.  The 
subsequent  instrument,  named  the  ALOQ,  was  administered  to  a  sample  of  the  target 
population.  Statistical  analysis  was  conducted,  examining  the  internal  consistency 
(Cronbach  alpha's)  of  the  ALOQ  and  it  was  found  to  have  satisfactory  level  of  statistic 
reliability. 

The  theoretical  validity  of  the  ALOQ  was  examined  using  the  nomological  process 
described  by  Babbie  (1995)  and  Streiner  and  Norman  (2003,  2008).  The  ALOQ  was 
developed  from  instruments  found  to  be  firmly  anchored  within  the  theoretical 
underpinnings  of  a  Learning  Organisation.  The  exploratory  factor  analysis  evaluated  the 
underlying  commonality  within  the  items,  and  found  a  very  similar  structure  to  those 
proposed  by  both  Marsick  and  Watkins  (2003)  and  Goh  and  Richards  (1997).  The  447  items 
derived  from  the  DLOQ  loaded  onto  eight  factors,  rather  than  Marsick  and  Watkin's  (2003) 
original  seven  factors.  The  DLOQ  factor  "creating  continuous  learning  opportunities"  was 
found,  within  the  Army  sample,  to  load  onto  two  separate  factors.  These  were  "resources 
and  rewards"  and  "learning  practices".  This  difference  to  Marsick  and  Watkins'  (2003) 
findings  may  reflect  the  centralisation  of  the  Army  as  an  organisation,  particularly  in 
terms  of  control  over  resource  allocation.  All  the  other  factors  described  by  Marsick  and 
Watkins'  (2003)  and  Goh  and  Richard's  (1997)  were  found  within  our  sample.  These 
finding  show  that  the  ALOQ  does  measure  the  underpinning  factors  within  an 
organisation  that  determine  the  quality  of  learning  within  organisations. 

In  conclusion,  the  ALOQ  was  designed  to  measure  Learning  Organisation  characteristics 
within  the  Australian  Army.  The  ALOQ  was  adapted  from  the  most  reliable  and  valid 
measures  of  learning  organisations  available  in  the  literature,  the  DLOQ  and  OLS.  The 
ALOQ  underwent  a  rigorous  process  of  testing  and  evaluation  to  ensure  its  reliability  and 
validity.  Using  data  collected  by  the  ALOQ,  we  can  determine  measures  the  underlying 
factors  that  influence  learning  within  the  Army  and  the  information  to  underpin  decisions 
relating  to  organisational  change. 


1  The  original  43  DLOQ  items  plus  an  additional  Army-specific  question  makes  up  the  44  item  set. 
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Appendix  A:  ALOQ  EFA  results 


A.l.  ALOQ  items 

Table  9:  ALOQ  44  item  and  21  item  dataset 


Construct 

Item# 

Item 

Create  continuous 
learning  opportunities 

1 

We  openly  discuss  mistakes  in  order  to  learn  from  them 

2 

We  Identify  skills  we  need  for  future  work  tasks 

3 

We  help  each  other  learn 

4 

We  can  get  money  and  other  resources  to  support  our  learning 

5 

We  are  given  time  to  learn 

6 

We  view  problems  in  our  work  as  an  opportunity  to  learn 

7 

We  are  rewarded  for  learning 

Promote  inquiry  and 
dialogue 

8 

We  give  open  and  honest  feedback  to  each  other 

9 

We  listen  to  others'  views  before  speaking 

10 

We  are  encouraged  to  ask  'why'  regardless  of  rank 

11 

Whenever  we  state  our  views  we  also  ask  what  others  think 

12 

We  treat  each  other  with  respect 

13 

We  spend  our  time  building  trust  with  each  other 

Encourage  collaboration 
and  team  learning 

14 

Workgroups/ Sections  have  the  freedom  to  change  their  goals  as  needed 

15 

Workgroups/ Sections  treat  members  as  equals  regardless  of  rank,  culture  or  any  other  differences 

16 

Workgroups/ Sections  focus  both  on  the  group's  task  and  on  how  well  the  group  is  working 

17 

Workgroups/ Sections  change  their  thinking  as  a  result  of  group  discussions 

18 

Workgroups/ Sections  are  rewarded  for  their  achievements  as  a  team  or  group 

19 

Workgroups/ Sections  are  confident  that  Army  will  act  on  their  suggestions 

Establish  systems  to 
capture  and  share 
learning 

20 

Army  uses  two-way  communications  regularly  (e.g.  suggestion  systems,  open  meetings) 

21 

Army  lets  us  get  the  information  required  to  do  our  jobs  quickly  and  easily 

22 

Army  maintains  an  up-to-date  profile  of  our  qualifications 

23 

Army  measures  gaps  between  current  and  expected  performance 

24 

Army  makes  lessons  learnt  available  to  all  its  people 

25 

Army  seeks  feedback  from  its  people  on  the  effectiveness  of  training  courses 

Empower  people  to  a 
collective  vision 

26 

Army  recognises  its  people  for  taking  initiative 

27 

Army  gives  its  people  choices  in  their  work  assignments 

28 

Army  asks  its  people  to  contribute  to  its  vision 

29 

Army  gives  its  people  control  over  the  resources  they  need  to  accomplish  their  work 

30 

Army  supports  people  with  innovative  work  practices 

31 

Army  allows  all  ranks  and  groups  to  have  a  say  in  Army's  vision 

Connect  the  organisation 
to  its  environment 

32 

Army  helps  its  people  to  balance  work  and  family  life 

33 

Army  encourages  its  people  to  think  about  the  global  perspective 

34 

Army  encourages  its  people  to  think  about  Army's  image  when  making  decisions 

35 

Army  considers  the  impact  of  decisions  on  morale 

36 

Army  works  with  communities  to  meet  both  Army's  and  communities'  needs 

37 

Army  encourages  its  people  to  seek  advice  from  different  parts  of  Army  when  solving  problems 

38* 

Army  allows  its  people  to  informally  discuss  problems  and  seek  advice  outside  their  unit* 

Provide  strategic 
leadership  for  learning 

39 

Supervisors  generally  support  requests  for  learning  opportunities  and  training 

40 

Supervisors  share  information  quickly  and  easily 

41 

Supervisors  empower  their  subordinates  to  help  carry  out  Army's  vision 

42 

Supervisors  mentor  and  coach  those  they  lead 

43 

Supervisors  continually  look  for  opportunities  to  learn 

44 

Supervisors  ensure  that  all  actions  are  consistent  with  Army's  values 

*Army-specific  item. 
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Construct 

Item# 

Item 

Leadership  commitment 
and  empowerment 

45 

Senior  leaders  resist  change  and  are  afraid  of  new  ideas 

46 

Senior  leaders  share  a  common  vision  with  each  other  of  what  our  work  should  accomplish 

47 

We  are  given  opportunities  to  provide  feedback  to  our  superiors 

48 

Supervisors  often  provide  useful  feedback  that  helps  to  identify  potential  problems  and  opportunities 

59 

Supervisors  frequently  involve  subordinates  in  important  decisions 

Experimentation  and 
rewards 

49 

Supervisors  encourage  team  members  to  experiment  in  order  to  improve  work  processes 

50 

Innovative  ideas  are  often  rewarded  by  supervisors 

51 

New  ideas  from  subordinates  are  not  treated  seriously  by  supervisors 

60 

I  often  bring  new  ideas  to  my  Workgroup/ Section 

61 

People  who  are  new  are  encouraged  to  question  the  way  things  are  done 

Transfer  of  knowledge 

52 

I  often  have  an  opportunity  to  talk  to  others  about  why  tasks  either  succeeded  or  failed 

53 

Failures  are  discussed  constructively 

54 

New  work  processes  that  may  be  useful  across  Army  are  shared  with  all  appropriate  workers 

55 

We  have  a  system  that  allows  us  to  learn  successful  practices  from  other  organisations 

Teamwork  and  group 
problem  solving 

56 

We  often  approach  our  supervisors  for  guidance  with  a  problem 

57 

We  can  usually  form  informal  groups  to  solve  problems 

58 

Most  problem-solving  features  people  from  a  variety  of  groups 

A.2.  ALOQ  communalities 

Table  10:  ALOQ  communalities  -  44  items  and  21  item  data  sets 


Initial 

Extraction 

Qi 

.505 

.530 

Q2 

.550 

.595 

Q3 

.522 

.551 

Q4 

.422 

.404 

Q5 

.486 

.490 

Q6 

.514 

.552 

Q  7 

.523 

.496 

Q8 

.484 

.505 

Q9 

.511 

.564 

Q10 

.462 

.488 

Qll 

.555 

.599 

Q12 

.559 

.566 

Q13 

.581 

.565 

Q14 

.477 

.472 

Q15 

.454 

.476 

Q16 

.611 

.666 

Q17 

.546 

.579 

Q18 

.546 

.549 

Q19 

.547 

.566 

Q20 

.503 

.493 

Q21 

.539 

.547 

Q22 

.388 

.452 

Initial 

Extraction 

Q23 

.492 

.547 

Q24 

.479 

.520 

Q25 

.397 

.404 

Q26 

.605 

.575 

Q27 

.546 

.569 

Q28 

.525 

.510 

Q29 

.540 

.541 

Q30 

.616 

.616 

Q31 

.510 

.510 

Q32 

.432 

.401 

Q33 

.491 

.503 

Q34 

.368 

.442 

Q35 

.538 

.567 

Q36 

.494 

.556 

Q37 

.584 

.580 

Q38 

.502 

.467 

Q39 

.553 

.550 

Q40 

.621 

.649 

Q41 

.633 

.654 

Q42 

.682 

.753 

Q43 

.686 

.726 

Q44 

.587 

.616 
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Initial 

Extraction 

Q45 

.215 

.187 

Q46 

.231 

.217 

Q47 

.516 

.560 

Q48 

.576 

.602 

Q59 

.476 

.490 

Q49 

.553 

.595 

Q50 

.521 

.562 

Q51 

.216 

.180 

Q60 

.168 

.234 

Initial 

Extraction 

Q61 

.349 

.342 

Q52 

.445 

.434 

Q53 

.542 

.543 

Q54 

.531 

.695 

Q55 

.499 

.580 

Q56 

.379 

.394 

Q57 

.413 

.474 

Q58 

.374 

.412 

Table  11:  Variance  explained  (Eigen  values)  for  44  item  data  set 


Factor 

Initial 

Eigenvalues 

%of 

Variance 

Cumulative 

% 

1 

18.921 

43.002 

43.002 

2 

2.095 

4.761 

47.763 

3 

1.693 

3.847 

51.610 

4 

1.290 

2.932 

54.542 

5 

1.103 

2.507 

57.049 

6 

1.066 

2.423 

59.472 

7 

.958 

2.177 

61.649 

8 

.869 

1.976 

63.625 

9 

.801 

1.820 

65.445 

10 

.708 

1.609 

67.054 

11 

.702 

1.595 

68.649 

12 

.679 

1.543 

70.192 

13 

.663 

1.506 

71.699 

14 

.618 

1.405 

73.104 

15 

.584 

1.327 

74.431 

16 

.547 

1.243 

75.674 

17 

.546 

1.240 

76.914 

18 

.519 

1.180 

78.094 

19 

.501 

1.139 

79.233 

20 

.485 

1.101 

80.334 

A.3.  EFA  results 

The  results  from  the  EFA  shows  that  the  44  interrelated  variables  can  be  reduced  to  eight 
factors,  rather  than  the  seven  hypothesised  by  Mar  sick  and  Watkins  (2003).  The  results  of 
pattern  matrix  suggest  the  following: 

1.  The  first  of  Marsick  and  Watkins'  (2003)  constructs  ("Create  continuous  learning 
opportunities")  is  comprised  of  two  sub-components,  namely,  Ql,  Q2,  Q3  and  Q6 
which  load  onto  Factor  3  in  the  Pattern  Matrix  and  Q4,  Q5  and  Q 7  which  load  onto 
Factor  8  in  the  Pattern  Matrix.  Items  Ql,  Q2,  Q3  &  Q6  reflect  "Learning  practices"  (Ql: 
"we  openly  discuss  mistakes  ...";  Q2:  "We  identify  skills  we  need  ...";  Q3:  "We  help 
each  other  to  learn";  Q6:  "We  view  problems  ...  as  opportunities  to  learn")  whereas 
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Q4,  Q5  &  Q7  reflect  perceptions  of  "Resourcing  &  Rewards"  (Q4:  "We  get  money  and 
other  resources  to  support  our  learning";  Q5:  "We  are  given  time  to  learn";  Q7:  "We 
are  rewarded  for  learning").  These  items  will  probably  need  to  be  treated  as  two 
separate  factors,  namely,  "Learning  Practices"  (Ql,  Q2,  Q3  &  Q6)  and  "Resourcing 
and  Rewards"  (Q4,  Q5  &  Q7). 

2.  Items  Q8  -  Q13  load  onto  one  factor  (Factor  4  in  the  Pattern  Matrix)  which  supports  the 
existence  of  Marsick  and  Watkins'  (20030  second  construct  ("Promote  inquiry  and 
dialogue").  Note,  however,  that  Q8  and  Q10  load  only  weakly  on  the  factor  and  a 
reading  of  Q12  and  Q13  suggest  that  these  items  are  different  to  the  others  four.  The 
weak  loading  for  Q10  ("We  are  encouraged  to  ask  'why'  regardless  of  rank")  is  not 
surprising  given  that  most  respondents  indicated  that  this  "Almost  never"  or  "Rarely" 
occurred  which  was  in  contrast  to  the  positive  "Often"  and  "Very  often"  responses  to 
the  other  five  items  measuring  this  construct. 

3.  The  third  of  Marsick  and  Watkins'  (2003)  constructs  ("Encourage  collaboration  and 
team  learning")  were  measured  by  items  Q14  -  Q19.  Elowever,  these  items  do  not  all 
load  onto  a  single  factor.  Items  Q15,  Q16  &  Q17  load  strongly  onto  one  factor  (Factor  7 
in  the  Pattern  Matrix)  and  these  are  most  closely  associated  with  collaboration  and 
team  learning  (Q15:  Workgroups  treat  members  as  equals  ...";  Q16:  "Workgroups 
focus  ...  on  how  well  the  group  is  working";  Q17:  Workgroups  change  their  thinking 
as  a  result  of  group  discussion").  Item  Q14  loads  on  none  of  the  factors  and  should 
probably  be  dropped.  Item  Q18,  which  talks  about  "rewards",  loads  more  on  Factor  8 
in  the  Pattern  Matrix  which  was  the  "Resourcing  and  rewards"  factor  which  includes 
Q7:  "We  are  rewarded  for  learning".  Item  Q19,  which  talks  about  Army  acting  on  the 
work-groups'  suggestions,  loads  more  on  Factor  1  in  the  Pattern  Matrix  which  is  about 
Army's  collaborative  vision. 

4.  The  fourth  of  Marsick  and  Watkins'  (2003)  constructs  ("Establish  systems  to  capture 
and  share  learning")  were  supposedly  measured  by  items  Q20  -  Q25.  Reading  of  the 
items  suggests  two  aspects  of  the  factor.  Items  Q22,  Q23  and  Q25  are  about  systems 
Army  has  in  place  to  track  learning  (Q22:  "Army  maintains  an  up-to-date  profile  of 
qualifications";  Q23:  Army  measures  gaps  between  current  and  expected 
performance";  Q25:  "Army  seeks  feedback  ...  on  the  effectiveness  of  training  courses") 
whereas  items  Q20,  Q21  and  Q24  are  about  ways  in  which  Army  shares  learning  (Q20: 
"Army  uses  two-way  communication  ...";  Q21:  Army  lets  us  get  the  information 
required  to  do  our  jobs  ...";  Q24:  "Army  makes  lessons  learnt  available  to  all  its 
people"). 

5.  Items  Q26  -  Q31  load  onto  one  factor  (Factor  1  in  the  Pattern  Matrix)  which  supports 
the  existence  of  Marsick  and  Watkins'  (2003)  fifth  construct  ("Empower  people 
towards  a  collaborative  vision").  Items  Q26,  Q29  and  Q30  are  about  recognising 
initiative  and  supporting  innovation  (Q26:  "Army  recognises  its  people  for  taking 
initiative";  Q29:  Army  gives  its  people  control  ...  to  accomplish  their  work";  Q25: 
"Army  supports  people  with  innovative  work  practices")  whereas  items  Q28  and  Q31 
are  about  Army's  collective  vision  (Q20:  "Army  asks  its  people  to  contribute  to  its 
vision";  Q31:  Army  allows  all  ..  to  have  a  say  in  Army's  vision").  Remember  also  that 
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item  Q19  loaded  onto  this  factor  and  is  related  to  a  collective  vision  in  that  it  talks 
about  Army  acting  on  suggestions.  Item  Q27  is  very  different  in  that  it  talks  about 
choice  in  work  assignments. 

6.  Items  Q32  -  Q38  load  onto  one  factor  (Factor  5  in  the  Pattern  Matrix)  which  supports 
the  existence  of  Marsick  and  Watkins'  (2003)  sixth  construct  ("Connect  the  organisation 
to  its  environment").  However,  a  close  reading  of  the  items  suggests  two  factors  with 
one  item  needing  to  be  dropped.  Items  Q33,  Q34,  Q35  and  Q36  are  about  Army's 
concerns  with  connecting  to  the  external  world  (Q33:  "Army  encourages  its  people  to 
think  about  the  global  perspective";  Q34:  Army  encourages  its  people  to  think  about 
Army's  image  ...";  Q36:  "Army  works  with  communities  to  meet  both  Army's  and 
communities'  needs")  whereas  items  Q37  and  Q38  are  about  seeking  external  advice 
(Q37:  "Army  encourages  its  people  to  seek  advice  from  different  parts  of  Army  ..."; 
Q38:  "Army  allows  its  people  to  informally  ...seek  advice  outside  of  their  unit").  Item 
Q58  from  the  Goh  and  Richards  (date)  "Learning  Capability  of  Organizations"  scale 
(Q58:  "Most  problem-solving  features  people  from  a  variety  of  groups")  also  appears 
to  be  closely  related  to  Items  Q37  and  Q38  and  may  need  to  be  included  here.  The 
weak  loading  for  item  Q32  ("Army  helps  its  people  balance  work  and  family  life")  is 
not  surprising  because  it  is  quite  different  in  nature  to  the  connectedness  to  the 
external  world  and  the  seeking  external  advice  items. 

7.  The  seventh  of  Marsick  and  Watkins'  (2003)  constructs  ("Provide  strategic  leadership 
for  learning")  also  appears  to  hold  items  Q39  -  Q44  all  loading  onto  one  factor  (Factor 
2  in  the  Pattern  Matrix).  However,  a  reading  of  the  items  suggests  two  aspects  of  the 
factor.  Items  Q42,  Q43  and  Q44  are  about  actions  supervisors  take  (Q42:  "Supervisors 
mentor  and  coach";  Q43:  Supervisors  continually  look  for  opportunities  to  learn";  Q44: 
"Supervisors  ensure  all  actions  are  consistent  with  Army's  values")  whereas  items 
Q39,  Q40  and  Q41  concern  subordinates  perception  about  how  they  are  lead  (Q39: 
"Supervisors  generally  support  requests  for  learning  opportunities  and  training";  Q40: 
"Supervisors  share  information  ...";  Q41:  "Supervisors  empower  their  subordinates  to 
carry  out  Army's  vision"). 

The  EFA  shows  that  the  relationships  amongst  the  21  item  data  set  variables  are  more 
complex  than  the  four  factor  solution  hypothesised  by  Goh  and  Richards.  An  initial  study 
of  the  pattern  matrix  suggests  that  items  Q45  and  Q51  (the  two  negatively  worded  items) 
load  onto  one  factor  (Factor  4  in  the  Pattern  Matrix).  This  factor  is  clearly  a  simple  method 
factor.  Given  that  the  communalities  for  these  two  items  were  extremely  low  it  is  clear  that 
these  two  items  consisted  of  measurement  error,  and  thus,  should  be  dropped  and  the 
analysis  re-run.  A  study  of  the  pattern  matrix  suggests  that  there  were  in  fact  four  factors: 

1.  Items  Q46  -  Q48  load  onto  one  factor  (Factor  1  in  the  Pattern  Matrix)  but  Q59 
which  is  also  meant  to  be  a  measure  of  "Leadership  Commitment  and 
Empowerment"  loads  on  a  different  factor.  Items  Q47,  Q48  and,  to  a  lesser  extent, 
Q46  are  about  feedback  up  to  and  back  from  superiors  (Q47:  "We  are  given 
opportunities  to  provide  feedback  to  superiors";  Q48:  "Superiors  often  provide 
useful  feedback  ...").  Item  59  is  quite  different  (Q59:  "Superiors  frequently  involve 
subordinates  in  important  decisions")  and  probably  should  be  omitted  from  this 
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factor.  Therefore,  three  of  these  items  will  probably  need  to  be  treated  as  a 
"Feedback"  factor  (Q46,  Q47  &  Q48)  with  the  possibility  of  Q59  loading  onto 
another  factor. 

2.  Although  the  Pattern  Matrix  suggests  that  item  Q49  and  Q50  load  onto  one  factor 
(Factor  6  in  the  Pattern  Matrix)  whereas  items  Q60  and  Q61  load  onto  another 
factor  (Factor  2  in  the  Pattern  Matrix),  a  reading  of  the  items  suggest  they  could  all 
be  part  of  an  "Experimentation"  factor  (Q49:  "Supervisors  encourage  team 
members  to  experiment  ...";  Q50:  "Innovative  ideas  are  often  rewarded  by 
superiors";  Q60:  "I  often  bring  new  ideas  ...";  Q61:  "People  ...  are  encourage  to 
question  the  way  things  are  done").  These  four  items  will  probably  need  to  be 
treated  as  an  "Experimentation"  factor  (Q49,  Q50,  Q60  &  Q61). 

3.  Again,  although  the  Pattern  Matrix  suggests  that  item  Q52  and  Q53  load  onto  one 
factor  (Factor  5  in  the  Pattern  Matrix)  whereas  items  Q54  and  Q55  load  onto 
another  factor  (Factor  3  in  the  Pattern  Matrix),  a  reading  of  the  items  suggest  they 
could  all  be  part  of  a  "Transfer  of  Knowledge"  factor  (Q52:  "I  often  have  an 
opportunity  to  talk  to  others  about  why  tasks  either  succeed  or  fail";  Q53:  "Failures 
are  discussed  constructively";  Q54:  "New  work  processors  . . .  are  shared  with  all 
appropriate  workers";  Q55:  "We  have  systems  that  allows  us  to  learn  successful 
practices  from  other  organisations").  These  four  items  will  probably  need  to  be 
treated  as  a  "Knowledge  Transfer"  factor  (Q52,  Q53,  Q54  &  Q55). 

4.  Again,  although  the  Pattern  Matrix  suggests  that  item  Q56  and  Q57  load  onto  one 
factor  (Factor  4  in  the  Pattern  Matrix)  whereas  items  Q58  loads  onto  another,  a 
reading  of  the  items  suggest  they  could  all  be  part  of  a  "Team  Problem-Solving" 
factor  (Q56:  "We  often  approach  our  supervisors  for  guidance  with  a  problem"; 
Q57:  "We  can  usually  form  informal  groups  to  solve  problems";  Q58:  "Most 
problem-solving  features  people  from  a  variety  of  groups").  These  three  items  will 
probably  need  to  be  treated  as  a  "Team  Problem-Solving"  factor  (Q56,  Q57  &  Q58). 
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